thetimes.com
robots.txt
Robots Exclusion Standard data for thetimes.com
Resource Scan
Scan Details
| Site Domain | thetimes.com |
| Base Domain | thetimes.com |
| Scan Status | Ok |
| Last Scan | 2024-06-30T23:44:43+00:00 |
| Next Scan | 2024-07-07T23:44:43+00:00 |
Last Scan
| Scanned | 2024-06-30T23:44:43+00:00 |
| URL | https://thetimes.com/robots.txt |
| Redirect | https://www.thetimes.com/robots.txt |
| Redirect Domain | www.thetimes.com |
| Redirect Base | thetimes.com |
| Domain IPs | 34.240.28.43, 52.208.17.106, 54.76.240.177 |
| Redirect IPs | 108.157.254.125, 108.157.254.4, 108.157.254.69, 108.157.254.93 |
| Response IP | 108.157.254.125 |
| Found | Yes |
| Hash | ccd2b234a8c7146e9763c15eb43e20b4ebaf74e53603e514e09b3fc768c96467 |
| SimHash | 3d50194b6fc6 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /login.thetimes.com/user/logout |
| Disallow | /feeds.thetimes.com/puzzles/ |
| Disallow | /feeds.thetimes.com/timescrossword/ |
| Disallow | /archive/page/* |
| Disallow | /archive/article/* |
| Disallow | /*?s=* |
| Disallow | /*%26s%3D* |
| Disallow | /*?p=* |
| Disallow | /*?filter=* |
| Allow | /past-six-days/$ |
| Allow | /past-six-days$ |
| Disallow | /past-six-days/* |
| Disallow | /topic/bbc |
| Disallow | /tto/* |
| Disallow | /player/brightcove/ |
| Disallow | /my-articles |
| Disallow | /my-articles/ |
| Disallow | /edition/null/ |
| Disallow | /goto |
| Disallow | /?region= |
| Disallow | /?_ga |
| Disallow | /?CMP |
| Disallow | /?ExternalDataReference |
| Disallow | /article/category/ |
| Disallow | /article/this-article-has-been-deleted* |
| Disallow | /article/this-article-has-been-removed* |
| Disallow | /article/this-article-is-no-longer-available* |
| Disallow | /search?* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.thetimes.com/sitemaps/sitemap.xml |
Comments