Log In Sign Up

slatecave.net
robots.txt

Robots Exclusion Standard data for slatecave.net

Resource Scan

Scan Details

Site Domain slatecave.net
Base Domain slatecave.net
Scan Status Ok
Last Scan2025-12-01T09:28:17+00:00
Next Scan 2025-12-31T09:28:17+00:00

Last Scan

Scanned2025-12-01T09:28:17+00:00
URL https://slatecave.net/robots.txt
Domain IPs 2a0a:4cc0:1:38c::1, 89.58.60.175
Response IP 89.58.60.175
Found Yes
Hash 5efbce879027289e13bcd678d26a08a27ec1aea654edb048a269d9f43215b3c5
SimHash 5a51bd602112

Groups

facebot

Rule Path
Disallow /

ioncrawl
piplbot

Rule Path
Disallow /

blexbot
ahrefsbot
mj12bot
barkrowler
semrushbot
dotbot
dataforseobot
serpstatbot

Rule Path
Disallow /

turnitinbot
npbot
slysearch
checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)
brandverity/1.0

Rule Path
Disallow /

chatgpt-user
gptbot
google-extended
anthropic-ai
claude-web
facebookbot
ai2bot

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://slatecave.net/sitemap.xml
Back to top

Comments

  • Note: this file is based on the robots.txt over at
  • https://seirdy.one/robots.txt
  • "Social" Media
  • Misc
  • Marketing or SEO
  • Intellectual "Property" scanners
  • AI-Training data scrapers
Back to top