Log In Sign Up

dexerto.fr
robots.txt

Robots Exclusion Standard data for dexerto.fr

Resource Scan

Scan Details

Site Domain dexerto.fr
Base Domain dexerto.fr
Scan Status Ok
Last Scan2026-03-09T20:12:30+00:00
Next Scan 2026-03-16T20:12:30+00:00

Last Scan

Scanned2026-03-09T20:12:30+00:00
URL https://dexerto.fr/robots.txt
Redirect https://www.dexerto.fr/robots.txt
Redirect Domain www.dexerto.fr
Redirect Base dexerto.fr
Domain IPs 104.26.8.94, 104.26.9.94, 172.67.68.241, 2606:4700:20::681a:85e, 2606:4700:20::681a:95e, 2606:4700:20::ac43:44f1
Redirect IPs 104.26.8.94, 104.26.9.94, 172.67.68.241, 2606:4700:20::681a:85e, 2606:4700:20::681a:95e, 2606:4700:20::ac43:44f1
Response IP 104.26.8.94
Found Yes
Hash eaab2d466a793e578671a63bc5f92b524ae74e0f977f1ac3a5549e66de69c55a
SimHash 611059d1e493

Groups

*

Rule Path
Disallow /search/
Disallow /cdn-cgi/
Allow /cdn-cgi/image/

amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
gptbot
httrack
nutch
offline explorer
scrapy
youbot
anthropic-ai
cohere-ai
omgili

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.dexerto.fr/sitemap_index.xml
sitemap https://www.dexerto.fr/news-sitemap.xml
Back to top

Comments

  • Block AI content scrapers
Back to top

Warnings

  • 1 invalid line.
Back to top