rostercon.com
robots.txt

Robots Exclusion Standard data for rostercon.com

Resource Scan

Scan Details

Site Domain rostercon.com
Base Domain rostercon.com
Scan Status Ok
Last Scan2024-09-21T01:09:38+00:00
Next Scan 2024-09-28T01:09:38+00:00

Last Scan

Scanned2024-09-21T01:09:38+00:00
URL https://rostercon.com/robots.txt
Domain IPs 109.234.164.57
Response IP 109.234.164.57
Found Yes
Hash 83b331f5a29d3455d594dba1f9d3fccb17873a64ae405bb5a70b09585dccc1b3
SimHash 03149c504f53

Groups

*

Rule Path
Disallow /wp-content/uploads/wp-import-export-lite/

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

archive.org_bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

soso spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

sogou spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

linguee bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.rostercon.com/sitemap.xml
sitemap https://www.rostercon.com/sitemap-news.xml

Comments

  • Ralentissement de robots