politico.com
robots.txt
Robots Exclusion Standard data for politico.com
Resource Scan
Scan Details
| Site Domain | politico.com |
| Base Domain | politico.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-11-02T00:42:09+00:00 |
| Next Scan | 2024-11-09T00:42:09+00:00 |
Last Successful Scan
| Scanned | 2024-02-23T10:22:42+00:00 |
| URL | https://politico.com/robots.txt |
| Redirect | https://www.politico.com:443/robots.txt |
| Redirect Domain | www.politico.com |
| Redirect Base | politico.com |
| Domain IPs | 18.214.93.144, 3.232.135.75, 44.215.200.138 |
| Redirect IPs | 104.18.41.251, 172.64.146.5, 2606:4700:4400::6812:29fb, 2606:4700:4400::ac40:9205 |
| Response IP | 104.18.41.251 |
| Found | Yes |
| Hash | 677e8e9421fb802ce99989e936e74e7abf66813ea5913467e19ef8faed0b10b3 |
| SimHash | 683505c021b2 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /2step |
| Disallow | /campaigntrailnotused |
| Disallow | /CandidateRanking |
| Disallow | /candidates |
| Disallow | /CustomCompanionTest |
| Disallow | /debate |
| Disallow | /ryan |
| Disallow | /promotions |
| Disallow | /2014-election/results/mobile/iphone |
| Disallow | /2014-election/results/mobile/ipad |
| Disallow | /_preview |
| Disallow | /326home |
| Disallow | /327home |
| Disallow | /_styleguide |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.politico.com/sitemap.xml |
| sitemap | https://www.politico.com/news-sitemap.xml |
| sitemap | https://www.politico.com/2024-election/results/sitemap-2024-elections.xml |