Log In Sign Up

aditsystems.de
robots.txt

Robots Exclusion Standard data for aditsystems.de

Resource Scan

Scan Details

Site Domain aditsystems.de
Base Domain aditsystems.de
Scan Status Ok
Last Scan2024-09-30T07:04:40+00:00
Next Scan 2024-10-14T07:04:40+00:00

Last Scan

Scanned2024-09-30T07:04:40+00:00
URL https://aditsystems.de/robots.txt
Redirect https://www.aditsystems.de/robots.txt
Redirect Domain www.aditsystems.de
Redirect Base aditsystems.de
Domain IPs 185.115.179.146, 2a02:74a0:a008:2015:f816:3eff:fe27:4bc
Redirect IPs 185.115.179.146, 2a02:74a0:a008:2015:f816:3eff:fe27:4bc
Response IP 185.115.179.146
Found Yes
Hash fc8f26472a29092075f5ccbb2f1809b70d8c5c728ebb9de9e56a18ba9dbe2da4
SimHash 335c97040c58

Groups

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /
Back to top

Comments

  • Code: https://github.com/ellie/notes
  • Source: https://darkvisitors.com/
  • OpenAI, ChatGPT
  • https://platform.openai.com/docs/gptbot
  • Google AI (Bard, etc)
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • Block common crawl
  • I have mixed feelings on this one, but many models are trained on this data
  • It is also used to bootstrap new search indices though
  • https://commoncrawl.org/ccbot
  • Facebook
  • https://developers.facebook.com/docs/sharing/bot/
  • Cohere.ai
  • https://darkvisitors.com/agents/cohere-ai
  • Perplexity
  • https://docs.perplexity.ai/docs/perplexitybot
  • Anthropic
  • https://darkvisitors.com/agents/anthropic-ai
  • ...also anthropic
  • https://darkvisitors.com/agents/claudebot
Back to top