charlieintel.com
robots.txt

Robots Exclusion Standard data for charlieintel.com

Resource Scan

Scan Details

Site Domain charlieintel.com
Base Domain charlieintel.com
Scan Status Ok
Last Scan2026-02-04T10:05:45+00:00
Next Scan 2026-02-11T10:05:45+00:00

Last Scan

Scanned2026-02-04T10:05:45+00:00
URL https://charlieintel.com/robots.txt
Redirect https://www.charlieintel.com/robots.txt
Redirect Domain www.charlieintel.com
Redirect Base charlieintel.com
Domain IPs 104.26.0.193, 104.26.1.193, 172.67.73.42, 2606:4700:20::681a:1c1, 2606:4700:20::681a:c1, 2606:4700:20::ac43:492a
Redirect IPs 104.26.0.193, 104.26.1.193, 172.67.73.42, 2606:4700:20::681a:1c1, 2606:4700:20::681a:c1, 2606:4700:20::ac43:492a
Response IP 104.26.0.193
Found Yes
Hash 2a88ac16c1683b769d93b804918fa4ab7a8aa281dfd387038939b13a11a9781e
SimHash 61185951e511

Groups

*

Rule Path
Disallow /cdn-cgi/
Allow /cdn-cgi/image/

amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
gptbot
httrack
nutch
offline explorer
scrapy
youbot
anthropic-ai
cohere-ai
omgili

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.charlieintel.com/sitemap_index.xml
sitemap https://www.charlieintel.com/news-sitemap.xml

Comments

  • Block AI content scrapers

Warnings

  • 1 invalid line.