reviewed.usatoday.com
robots.txt
Robots Exclusion Standard data for reviewed.usatoday.com
Resource Scan
Scan Details
| Site Domain | reviewed.usatoday.com |
| Base Domain | usatoday.com |
| Scan Status | Ok |
| Last Scan | 2024-04-27T07:27:50+00:00 |
| Next Scan | 2024-05-04T07:27:50+00:00 |
Last Scan
| Scanned | 2024-04-27T07:27:50+00:00 |
| URL | https://reviewed.usatoday.com/robots.txt |
| Domain IPs | 151.101.130.62, 151.101.194.62, 151.101.2.62, 151.101.66.62 |
| Response IP | 199.232.46.62 |
| Found | Yes |
| Hash | ce9927cf0be1693be5a628fa3493ae44afef4285d36ce3c912b8df9911b955be |
| SimHash | ea44c831e7a0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /affiliates/ |
| Disallow | /shopping/ |
| Disallow | /featured/ |
| Disallow | /c/ |
| Disallow | /a/ |
| Disallow | /t/ |
| Disallow | /*.htm.amp |
| Disallow | /*?preview=true$ |
| Disallow | /.well-known/ |
| Disallow | /*.amp.ampAs |
| Disallow | /.ampAs |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://reviewed-misc.s3.amazonaws.com/sitemaps-new/sitemap.xml.gz |
Warnings
- 2 invalid lines.