metro.co.uk
robots.txt
Robots Exclusion Standard data for metro.co.uk
Resource Scan
Scan Details
| Site Domain | metro.co.uk |
| Base Domain | metro.co.uk |
| Scan Status | Ok |
| Last Scan | 2024-04-27T12:54:49+00:00 |
| Next Scan | 2024-05-04T12:54:49+00:00 |
Last Scan
| Scanned | 2024-04-27T12:54:49+00:00 |
| URL | https://metro.co.uk/robots.txt |
| Domain IPs | 192.0.66.24 |
| Response IP | 192.0.66.24 |
| Found | Yes |
| Hash | 5d0f317fbc0a6edcf622d682bb529e7b60e34a4fe6a2e76df15e8d073586834a |
| SimHash | 012088a08d9a |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Allow | /wp-admin/admin-ajax.php |
| Disallow | /search/ |
| Disallow | |
| Disallow | /post-style/* |
| Disallow | /jobs-scrape/* |
| Disallow | /*ia_markup%3D* |
| Disallow | /dmg_liveblog/* |
Other Records
| Field | Value |
|---|---|
| sitemap | http://metro.co.uk/news-sitemap.xml |
| sitemap | https://metro.co.uk/sitemap.xml?yyyy=2023 |
| sitemap | https://metro.co.uk/sitemap.xml?yyyy=2022 |
| sitemap | https://metro.co.uk/sitemap.xml?yyyy=2021 |
| sitemap | https://metro.co.uk/sitemap.xml?yyyy=2020 |
| sitemap | https://metro.co.uk/news-sitemap.xml |
| sitemap | https://metro.co.uk/sitemap.xml |
| sitemap | https://metro.co.uk/sitemap-channels.xml |
| sitemap | https://metro.co.uk/sitemap-tags.xml |
| sitemap | https://metro.co.uk/sitemap-authors.xml |
| sitemap | https://metro.co.uk/sitemap-videos.xml |
| sitemap | https://metro.co.uk/sitemap-tv-soaps.xml |
| sitemap | https://metro.co.uk/sitemap-images.xml |
| sitemap | https://metro.co.uk/sitemap.xml |
Comments