free.fr
robots.txt

Robots Exclusion Standard data for free.fr

Resource Scan

Scan Details

Site Domain free.fr
Base Domain free.fr
Scan Status Ok
Last Scan2024-06-15T03:11:59+00:00
Next Scan 2024-06-29T03:11:59+00:00

Last Scan

Scanned2024-06-15T03:11:59+00:00
URL https://free.fr/robots.txt
Redirect https://www.free.fr/robots.txt
Redirect Domain www.free.fr
Redirect Base free.fr
Domain IPs 212.27.48.10, 2a01:e0c:1::1
Redirect IPs 212.27.48.10, 2a01:e0c:1::1
Response IP 212.27.48.10
Found Yes
Hash 13673666e0fa08e8f600fa07bb77a7a4bb1cee2cd964dbbf6a2e3f7d9dff2c7e
SimHash 211c76c1c70b

Groups

ia_archiver
scoutjet
facebot
searchmetricsbot
turnitinbot
httrack

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /apps/
Disallow /*contentOnly*
Disallow

Other Records

Field Value
sitemap https://www.free.fr/sitemap.xml

Comments

  • Instructions sur les badbots
  • Alexa Archive, IBM Watson, Facebook, Search metrics
  • Linux, Base de donnée étudiante, Aspirateur web
  • Instructions pour les autres bots