pitandquarry.com
robots.txt

Robots Exclusion Standard data for pitandquarry.com

Resource Scan

Scan Details

Site Domain pitandquarry.com
Base Domain pitandquarry.com
Scan Status Ok
Last Scan2024-10-25T17:14:17+00:00
Next Scan 2024-11-24T17:14:17+00:00

Last Scan

Scanned2024-10-25T17:14:17+00:00
URL https://pitandquarry.com/robots.txt
Redirect https://www.pitandquarry.com/robots.txt
Redirect Domain www.pitandquarry.com
Redirect Base pitandquarry.com
Domain IPs 162.159.135.42
Redirect IPs 162.159.135.42
Response IP 162.159.135.42
Found Yes
Hash 8c9e2e3b25f5c4a06ff10ab1f1a6b94cd0ec7b08523680f40903d9f5e120d7fc
SimHash 601ec9c0e4d5

Groups

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

Comments

  • Additions to block AI scrapers