donmccurdy.com
robots.txt

Robots Exclusion Standard data for donmccurdy.com

Resource Scan

Scan Details

Site Domain donmccurdy.com
Base Domain donmccurdy.com
Scan Status Ok
Last Scan2024-11-01T04:32:01+00:00
Next Scan 2024-12-01T04:32:01+00:00

Last Scan

Scanned2024-11-01T04:32:01+00:00
URL https://donmccurdy.com/robots.txt
Redirect https://www.donmccurdy.com/robots.txt
Redirect Domain www.donmccurdy.com
Redirect Base donmccurdy.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.61, 76.76.21.9
Response IP 76.76.21.22
Found Yes
Hash 6ca943b750d2f7b5ed4d33bec67872c640425bbfb4cb2842fffccc8d2f313671
SimHash 32b459420127

Groups

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

Comments

  • References:
  • - https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/