agefi.fr
robots.txt

Robots Exclusion Standard data for agefi.fr

Resource Scan

Scan Details

Site Domain agefi.fr
Base Domain agefi.fr
Scan Status Ok
Last Scan2024-06-24T01:42:17+00:00
Next Scan 2024-07-01T01:42:17+00:00

Last Scan

Scanned2024-06-24T01:42:17+00:00
URL https://agefi.fr/robots.txt
Redirect https://www.agefi.fr/robots.txt
Redirect Domain www.agefi.fr
Redirect Base agefi.fr
Domain IPs 62.23.155.217
Redirect IPs 13.225.4.109, 13.225.4.48, 13.225.4.70, 13.225.4.98
Response IP 13.225.4.109
Found Yes
Hash 35f51efd1afb5f26889d0f01298fc51e8cb72a573821eabe90dad2c572a0003f
SimHash 4404e961e411

Groups

*

Rule Path
Disallow /search
Disallow /_auth/
Disallow /_track
Disallow /auteur/null*

linkedinbot

Rule Path
Allow /

amazonbot
anthropic-ai
applebot
bytespider
ccbot
chatgpt-user
cohere-ai
diffbot
facebookbot
friendlycrawler
google-extended
gptbot
omgili
perplexitybot
youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.agefi.fr/sitemap.xml
sitemap https://www.agefi.fr/news-sitemap.xml

Comments

  • AI crawlers