Log In Sign Up

luisroc.com
robots.txt

Robots Exclusion Standard data for luisroc.com

Resource Scan

Scan Details

Site Domain luisroc.com
Base Domain luisroc.com
Scan Status Ok
Last Scan2025-09-29T22:12:00+00:00
Next Scan 2025-10-29T22:12:00+00:00

Last Scan

Scanned2025-09-29T22:12:00+00:00
URL https://luisroc.com/robots.txt
Domain IPs 185.158.133.1
Response IP 185.158.133.1
Found Yes
Hash 8dfe33cbcb3008e07e44a1bfb785b5e3cd7d6446174ff3660845bb12c06ccbaf
SimHash 44091a50c4f0

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

gptbot

Rule Path
Allow /

openai-searchbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

bard

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

*

Rule Path
Allow /
Disallow /admin
Disallow /auth
Disallow /admin-setup

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.luisroc.com/sitemap.xml
Back to top

Comments

  • Robots.txt optimized for LLM crawlers and AI search engines
  • Luis ROC - SEO Consultant & LLM Positioning Expert
  • https://www.luisroc.com
  • Standard search engines
  • AI crawlers and LLM bots
  • All other bots
  • Block admin areas only
  • Sitemap location for AI crawlers
  • Crawl delay for respectful crawling
Back to top