Log In Sign Up

terrybicycles.com
robots.txt

Robots Exclusion Standard data for terrybicycles.com

Resource Scan

Scan Details

Site Domain terrybicycles.com
Base Domain terrybicycles.com
Scan Status Ok
Last Scan2024-10-26T02:04:48+00:00
Next Scan 2024-11-25T02:04:48+00:00

Last Scan

Scanned2024-10-26T02:04:48+00:00
URL https://www.terrybicycles.com/robots.txt
Domain IPs 23.215.7.16, 23.215.7.22
Response IP 23.32.29.106
Found Yes
Hash 0a0ad9fafee07eccdfff8a457fc51eb3397a2f94a239c23304569673fcf94ed4
SimHash 47b4de72ed23

Groups

*

Rule Path
Disallow /app/

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Disallow /app/
Disallow /api/
Disallow *page%3D
Disallow /product/
Disallow /itemtype/

googlebot-image

Rule Path
Disallow

gptbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://a335828.sitemaphosting7.com/4406513/sitemap_4406513.xml
Back to top

Comments

  • Allow all robots to spider everything by disallowing nothing
  • added new-sitemap as a temp solution
Back to top