Log In Sign Up

motorcyclecruiser.com
robots.txt

Robots Exclusion Standard data for motorcyclecruiser.com

Resource Scan

Scan Details

Site Domain motorcyclecruiser.com
Base Domain motorcyclecruiser.com
Scan Status Ok
Last Scan2024-11-14T12:49:26+00:00
Next Scan 2024-11-21T12:49:26+00:00

Last Scan

Scanned2024-11-14T12:49:26+00:00
URL https://motorcyclecruiser.com/robots.txt
Redirect https://www.motorcyclecruiser.com:443/robots.txt
Redirect Domain www.motorcyclecruiser.com
Redirect Base motorcyclecruiser.com
Domain IPs 15.197.174.213, 3.33.166.34
Redirect IPs 23.46.230.151, 23.46.230.153, 2600:1413:b000:13::b857:c18e, 2600:1413:b000:13::b857:c196
Response IP 23.45.207.171
Found Yes
Hash 2c5cab06a3ea2566f55b2640fce3528615fd393c99754980dc4624898eda5bc1
SimHash ac14cae22503

Groups

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

*

Rule Path
Disallow /au/
Disallow /ca/
Disallow /fr/
Disallow /ca/
Disallow /fr/
Disallow /de/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /mx/
Disallow /es/
Disallow /uk/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.motorcyclecruiser.com/arcio/sitemap-index/index/
sitemap https://www.motorcyclecruiser.com/arcio/fronts-sitemap/
Back to top

Comments

  • Disallow the following spiders
Back to top