Log In Sign Up

nikeshoebot.com
robots.txt

Robots Exclusion Standard data for nikeshoebot.com

Resource Scan

Scan Details

Site Domain nikeshoebot.com
Base Domain nikeshoebot.com
Scan Status Ok
Last Scan2025-11-27T15:33:04+00:00
Next Scan 2025-12-27T15:33:04+00:00

Last Scan

Scanned2025-11-27T15:33:04+00:00
URL https://nikeshoebot.com/robots.txt
Domain IPs 190.92.152.185
Response IP 190.92.152.185
Found Yes
Hash eb4d42e2502a70eb10606db620926d1c13524d84beb5649fdc0371331100c13a
SimHash 41b2d96e0d23

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /?s=
Disallow /search/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 10

google-extended

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

barkrowler

Rule Path
Allow /

ccbot

Rule Path
Allow /
Back to top

Comments

  • General crawler rules
  • ----------------------------
  • Allow access for AI crawlers
  • ----------------------------
  • Google’s AI (Gemini / AI Overviews)
  • OpenAI’s ChatGPT Browse / o1-preview models
  • Anthropic Claude’s AI crawler
  • Perplexity AI
  • Brave Search’s AI crawler
  • Common Crawl (used by many LLMs for pretraining)
Back to top