Log In Sign Up

superwebpros.com
robots.txt

Robots Exclusion Standard data for superwebpros.com

Resource Scan

Scan Details

Site Domain superwebpros.com
Base Domain superwebpros.com
Scan Status Ok
Last Scan2025-10-06T00:35:05+00:00
Next Scan 2025-11-05T00:35:05+00:00

Last Scan

Scanned2025-10-06T00:35:05+00:00
URL https://superwebpros.com/robots.txt
Domain IPs 104.21.33.221, 172.67.193.29, 2606:4700:3033::6815:21dd, 2606:4700:3034::ac43:c11d
Response IP 104.21.33.221
Found Yes
Hash 1ff1b2cf2a3e668dc9d565be2e7863707db9f2aae076adf5b7bf7deab8ecb1a4
SimHash c0409b618da3

Groups

*

Rule Path
Allow /
Disallow /chat
Disallow /c/
Disallow /api/chat
Disallow /api/c/
Disallow /api/conversations/
Disallow /admin
Disallow /preview/
Disallow /video-test
Disallow /demo
Allow /articles/
Allow /episodes/
Allow /
Allow /llms.txt
Back to top

Other Records

Field Value
sitemap https://superwebpros.com/sitemap.xml
Back to top

Comments

  • SuperWebPros - Robots.txt
  • Allow general crawling but protect chat/conversation routes
  • Block chat and conversation routes from indexing
  • Block admin and preview routes
  • Block development and testing routes
  • Allow important content for SEO
  • Sitemap location
  • Allow AI agents and crawlers to access llms.txt
Back to top