Log In Sign Up

afup.org
robots.txt

Robots Exclusion Standard data for afup.org

Resource Scan

Scan Details

Site Domain afup.org
Base Domain afup.org
Scan Status Ok
Last Scan2025-11-28T09:30:16+00:00
Next Scan 2025-12-28T09:30:16+00:00

Last Scan

Scanned2025-11-28T09:30:16+00:00
URL https://afup.org/robots.txt
Domain IPs 91.208.207.214, 91.208.207.215, 91.208.207.216, 91.208.207.217, 91.208.207.218, 91.208.207.220, 91.208.207.221, 91.208.207.222, 91.208.207.223
Response IP 91.208.207.223
Found Yes
Hash 2377a5b286df28d0efc430bd2bb82ebffce684e58a57cf6da28e048635cf1d3f
SimHash f49c10308c02

Groups

gptbot
chatgpt-user
oai-searchbot
google-extended
applebot-extended
anthropic-ai
claudebot
claude-web
amazonbot
cohere-ai
perplexitybot
youbot
ccbot
omgilibot
omgili
webzio-extended
facebookbot
meta-externalagent
bytespider
ai2bot
ai2bot-dolma
diffbot
pangubot
petalbot
timpibot

Rule Path
Disallow /
Back to top

Comments

  • OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT
  • https://platform.openai.com/docs/bots
  • ChatGPT plugins
  • https://platform.openai.com/docs/bots
  • OpenAI Search bot
  • https://platform.openai.com/docs/bots
  • Google's AI crawler
  • https://blog.google/technology/ai/an-update-on-web-publisher-controls/
  • Apple's AI crawler
  • https://support.apple.com/en-us/119829
  • Anthropic AI (Claude)
  • https://darkvisitors.com/operators/anthropic
  • Amazonbot
  • https://developer.amazon.com/amazonbot
  • Cohere
  • Perplexity
  • You
  • https://about.you.com/fr/youbot/
  • Common Crawl
  • https://commoncrawl.org/ccbot
  • Omglibot: webz.io
  • https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/
  • Facebook: Llama
  • https://developers.facebook.com/docs/sharing/bot/
  • Facebook
  • https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  • ByteDance: Duobao
  • https://darkvisitors.com/operators/bytedance
  • Ai2
  • https://allenai.org/crawler
  • Diffbot
  • https://darkvisitors.com/agents/diffbot
  • Huawei
  • https://darkvisitors.com/agents/pangubot
  • Petal Search
  • https://datadome.co/learning-center/how-to-block-petal-bot/
  • Timpibot
  • https://darkvisitors.com/agents/timpibot
  • Target
Back to top