Log In Sign Up

motorcyclistonline.com
robots.txt

Robots Exclusion Standard data for motorcyclistonline.com

Resource Scan

Scan Details

Site Domain motorcyclistonline.com
Base Domain motorcyclistonline.com
Scan Status Ok
Last Scan2024-11-15T06:18:54+00:00
Next Scan 2024-11-22T06:18:54+00:00

Last Scan

Scanned2024-11-15T06:18:54+00:00
URL https://motorcyclistonline.com/robots.txt
Redirect https://www.motorcyclistonline.com:443/robots.txt
Redirect Domain www.motorcyclistonline.com
Redirect Base motorcyclistonline.com
Domain IPs 15.197.174.213, 3.33.166.34
Redirect IPs 184.87.193.85, 184.87.193.88, 2600:1413:b000:13::b857:c18e, 2600:1413:b000:13::b857:c196
Response IP 23.45.207.165
Found Yes
Hash 9ef77b0c0e68b3ff8c3d29dda78dad3ec6718a12b3772e78ecbe6a2a74ddcd59
SimHash ac46c8f28503

Groups

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

*

Rule Path
Disallow /au/
Disallow /ca/
Disallow /fr/
Disallow /ca/
Disallow /fr/
Disallow /de/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /mx/
Disallow /es/
Disallow /uk/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.motorcyclistonline.com/arcio/sitemap-index/index/
sitemap https://www.motorcyclistonline.com/arcio/fronts-sitemap/
Back to top

Comments

  • Disallow the following spiders
Back to top