Log In Sign Up

re-enthused.com
robots.txt

Robots Exclusion Standard data for re-enthused.com

Resource Scan

Scan Details

Site Domain re-enthused.com
Base Domain re-enthused.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-08T11:55:40+00:00
Next Scan 2025-11-07T11:55:40+00:00

Last Successful Scan

Scanned2025-09-14T22:28:07+00:00
URL https://re-enthused.com/robots.txt
Domain IPs 185.151.30.170, 2a07:7800::170
Response IP 185.151.30.170
Found Yes
Hash 889fd9cb2db454622fc32826c9189b8ae0f18f72bff246088b850c360005fe24
SimHash 337c97040c78

Groups

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /
Back to top

Comments

  • OpenAI, ChatGPT
  • https://platform.openai.com/docs/gptbot
  • Google AI (Bard, etc)
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • Block common crawl
  • I have mixed feelings on this one, but many models are trained on this data
  • It is also used to bootstrap new search indices though
  • https://commoncrawl.org/ccbot
  • Facebook
  • https://developers.facebook.com/docs/sharing/bot/
  • Cohere.ai
  • https://darkvisitors.com/agents/cohere-ai
  • Perplexity
  • https://docs.perplexity.ai/docs/perplexitybot
  • Anthropic
  • https://darkvisitors.com/agents/anthropic-ai
  • ...also anthropic
  • https://darkvisitors.com/agents/claudebot
Back to top