20minutes.fr
robots.txt
Robots Exclusion Standard data for 20minutes.fr
Resource Scan
Scan Details
Site Domain | 20minutes.fr |
Base Domain | 20minutes.fr |
Scan Status | Ok |
Last Scan | 2024-04-24T04:38:16+00:00 |
Next Scan | 2024-05-01T04:38:16+00:00 |
Last Scan
Scanned | 2024-04-24T04:38:16+00:00 |
URL | https://20minutes.fr/robots.txt |
Redirect | https://www.20minutes.fr/robots.txt |
Redirect Domain | www.20minutes.fr |
Redirect Base | 20minutes.fr |
Domain IPs | 108.157.254.26, 108.157.254.42, 108.157.254.66, 108.157.254.9 |
Redirect IPs | 152.195.37.212 |
Response IP | 152.195.37.212 |
Found | Yes |
Hash | efc00443858ccce56cb47a175e728fa986fd655b0765cd8c4b839449555976b4 |
SimHash | 224f50504e05 |
Groups
*
Rule | Path |
---|---|
Disallow | /article/*/commentaires* |
Disallow | /resultats-examen/recherche/ |
Disallow | /resultats-examen/candidat/ |
Disallow | /embed/elections/resultats/ |
Disallow | /v-ajax |
Disallow | /v-esi |
Disallow | /search |
Other Records
Field | Value |
---|---|
sitemap | https://www.20minutes.fr/sitemap-arbo.xml |
Warnings
- 4 invalid lines.