Log In Sign Up

dirtrider.com
robots.txt

Robots Exclusion Standard data for dirtrider.com

Resource Scan

Scan Details

Site Domain dirtrider.com
Base Domain dirtrider.com
Scan Status Ok
Last Scan2024-11-15T08:12:19+00:00
Next Scan 2024-11-22T08:12:19+00:00

Last Scan

Scanned2024-11-15T08:12:19+00:00
URL https://dirtrider.com/robots.txt
Redirect https://www.dirtrider.com:443/robots.txt
Redirect Domain www.dirtrider.com
Redirect Base dirtrider.com
Domain IPs 15.197.174.213, 3.33.166.34
Redirect IPs 23.52.171.130, 23.52.171.161, 2600:1413:1::7d38:db10, 2600:1413:1::7d38:db1b
Response IP 23.45.207.175
Found Yes
Hash b9bf6799573b0357a3e14b69c25654210022f1726eaff623ac070796c40b9840
SimHash 8c06da702503

Groups

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

*

Rule Path
Disallow /au/
Disallow /ca/
Disallow /fr/
Disallow /ca/
Disallow /fr/
Disallow /de/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /mx/
Disallow /es/
Disallow /uk/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.dirtrider.com/arcio/sitemap-index/index/
sitemap https://www.dirtrider.com/arcio/fronts-sitemap/
Back to top

Comments

  • Disallow the following spiders
Back to top