gracoroberts.com
robots.txt

Robots Exclusion Standard data for gracoroberts.com

Resource Scan

Scan Details

Site Domain gracoroberts.com
Base Domain gracoroberts.com
Scan Status Ok
Last Scan2025-11-26T06:51:46+00:00
Next Scan 2025-12-26T06:51:46+00:00

Last Scan

Scanned2025-11-26T06:51:46+00:00
URL https://gracoroberts.com/robots.txt
Redirect https://www.gracoroberts.com/robots.txt
Redirect Domain www.gracoroberts.com
Redirect Base gracoroberts.com
Domain IPs 93.184.250.183
Redirect IPs 93.184.250.183
Response IP 93.184.250.183
Found Yes
Hash 62f22e803c7cb2200468800239d8a823e8cc779704d57f5a50acd321842c0ebb
SimHash 18765b16e013

Groups

chatgpt-user
claudebot
gptbot
oai-searchbot
ccbot
google-extended

Rule Path
Disallow

appinsights
awariobot
baiduspider
barkrowler
blexbot
brightedge crawler
dataforseobot
dotbot
geedobot
geedoproductsearch
gptbot
ioncrawl
megaindex
meta-externalagent
mj12bot
nexcess
politecrawl
qwantbot
seekport crawler
seznambot
stormcrawler
yandexbot

Rule Path
Disallow /

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /cart/
Disallow /login/
Disallow /search-results/
Disallow /checkout/
Disallow /my-account/
Disallow /*?*Product-Form=*
Disallow /*?*Brand=*
Disallow /*?*searchterm=*
Disallow /Services/*.asmx
Disallow /services/api/silmid/

Other Records

Field Value
sitemap https://www.gracoroberts.com/sitemap_index.xml

Comments

  • Allow AI crawlers
  • Disallow non-AI or unwanted crawlers
  • Throttle a specific bot
  • General rules for all crawlers