Log In Sign Up

adventuresmithexplorations.com
robots.txt

Robots Exclusion Standard data for adventuresmithexplorations.com

Resource Scan

Scan Details

Site Domain adventuresmithexplorations.com
Base Domain adventuresmithexplorations.com
Scan Status Ok
Last Scan2025-09-25T13:44:09+00:00
Next Scan 2025-10-25T13:44:09+00:00

Last Scan

Scanned2025-09-25T13:44:09+00:00
URL https://adventuresmithexplorations.com/robots.txt
Domain IPs 104.26.14.100, 104.26.15.100, 172.67.70.156, 2606:4700:20::681a:e64, 2606:4700:20::681a:f64, 2606:4700:20::ac43:469c
Response IP 104.26.15.100
Found Yes
Hash d2e5564d7fee58bc14aefe97cc608acad89b1f8fe67e11211a60987c9364b6e3
SimHash 7b2ad991ccf1

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /*?replytocom=
Disallow */feed/
Disallow */trackback/
Disallow */page/*
Allow /wp-content/uploads/

gptbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-user

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

amazonbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://adventuresmithexplorations.com/sitemap_index.xml
Back to top

Comments

  • AdventureSmith Explorations robots.txt — SEO-first + AI visibility (2025-09-19)
  • Notes:
  • - Keep render resources (CSS/JS) crawlable
  • - Disallow only low-value parameterized pages (search, replytocom)
  • - Point to the sitemap index (Yoast)
  • Keep admin private but allow Ajax for front-end features
  • De-prioritize thin/duplicate utility URLs
  • Uploads (images/docs) are explicitly allowed (not strictly required, but clear)
  • Sitemaps (index is sufficient)
  • AI / LLM crawlers — ALLOW (Option A)
Back to top