Log In Sign Up

apexpredators.com
robots.txt

Robots Exclusion Standard data for apexpredators.com

Resource Scan

Scan Details

Site Domain apexpredators.com
Base Domain apexpredators.com
Scan Status Ok
Last Scan2025-02-22T08:58:46+00:00
Next Scan 2025-03-24T08:58:46+00:00

Last Scan

Scanned2025-02-22T08:58:46+00:00
URL https://apexpredators.com/robots.txt
Domain IPs 104.21.51.168, 172.67.182.254, 2606:4700:3033::6815:33a8, 2606:4700:3033::ac43:b6fe
Response IP 104.21.51.168
Found Yes
Hash 1735d3465db56f511fa14479c44a2fd649bf208ac3ef38c5e71ce68faab96e70
SimHash 5a98d3102622

Groups

*

Rule Path
Allow /wp-content/uploads/

amazonbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

awariorssbot

Rule Path
Allow /

awariosmartbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

ccbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

claude

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-web

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

dataforseobot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

meta ai

Rule Path
Allow /

google-extended

Rule Path
Allow /

google gemini

Rule Path
Allow /

grok

Rule Path
Allow /

imagesiftbot

Rule Path
Allow /

magpie-crawler

Rule Path
Allow /

microsoft copilot

Rule Path
Allow /

omgili

Rule Path
Allow /

omgilibot

Rule Path
Allow /

peer39_crawler

Rule Path
Allow /

peer39_crawler/1.0

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

youbot

Rule Path
Allow /
Back to top

Comments

  • Amazon bots
  • Anthropic AI bots
  • Apple bots
  • Awario bots
  • ByteDance bot
  • Common Crawl bot
  • OpenAI and ChatGPT bots
  • Claude bots (Anthropic)
  • Cohere AI bot
  • DataForSeo bot
  • Meta (Facebook) bots
  • Google bots
  • Grok bot (Samsung)
  • Imagesift bot
  • Magpie bot
  • Microsoft bots
  • Omgili bots
  • Peer39 crawlers
  • Perplexity bot
  • You.com bot
Back to top