Log In Sign Up

fluentea.com
robots.txt

Robots Exclusion Standard data for fluentea.com

Resource Scan

Scan Details

Site Domain fluentea.com
Base Domain fluentea.com
Scan Status Ok
Last Scan2026-03-06T23:48:51+00:00
Next Scan 2026-04-05T23:48:51+00:00

Last Scan

Scanned2026-03-06T23:48:51+00:00
URL https://fluentea.com/robots.txt
Domain IPs 104.21.49.53, 172.67.159.10, 2606:4700:3034::ac43:9f0a, 2606:4700:3035::6815:3135
Response IP 172.67.159.10
Found Yes
Hash 35e69575f659ab7edeba48b8b348d84fb229510f9c9549fb747d0870578a1d7c
SimHash 68305016fa01

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /ws/
Disallow /cdn-cgi/

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://fluentea.com/sitemap.xml
Back to top

Comments

  • Allow all crawlers to access the site
  • Disallow access to API endpoints and admin areas
  • Disallow CDN and email protection URLs
  • Explicitly allow ChatGPT and OpenAI crawlers
  • Allow other AI crawlers
  • Sitemap location
Back to top