theguardian.com
robots.txt
Robots Exclusion Standard data for theguardian.com
Resource Scan
Scan Details
| Site Domain | theguardian.com |
| Base Domain | theguardian.com |
| Scan Status | Ok |
| Last Scan | 2024-10-31T23:15:54+00:00 |
| Next Scan | 2024-11-07T23:15:54+00:00 |
Last Scan
| Scanned | 2024-10-31T23:15:54+00:00 |
| URL | https://theguardian.com/robots.txt |
| Redirect | https://www.theguardian.com/robots.txt |
| Redirect Domain | www.theguardian.com |
| Redirect Base | theguardian.com |
| Domain IPs | 151.101.1.111, 151.101.129.111, 151.101.193.111, 151.101.65.111, 2a04:4e42:200::367, 2a04:4e42:400::367, 2a04:4e42:600::367, 2a04:4e42::367 |
| Redirect IPs | 151.101.1.111, 151.101.129.111, 151.101.193.111, 151.101.65.111, 2a04:4e42:200::367, 2a04:4e42:400::367, 2a04:4e42:600::367, 2a04:4e42::367 |
| Response IP | 199.232.45.111 |
| Found | Yes |
| Hash | 1355a03101eb6a27a14d0f9e840bbec3c3e45e4b7104b12a8442f34b569efab9 |
| SimHash | cf015509e7e6 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /sendarticle/ |
| Disallow | /Users/ |
| Disallow | /users/ |
| Disallow | /*/print$ |
| Disallow | /email/ |
| Disallow | /contactus/ |
| Disallow | /share/ |
| Disallow | /websearch |
| Disallow | /*?commentpage= |
| Disallow | /whsmiths/ |
| Disallow | /external/overture/ |
| Disallow | /discussion/report-abuse/* |
| Disallow | /discussion/report-abuse-ajax/* |
| Disallow | /discussion/comment-permalink/* |
| Disallow | /discussion/report-abuse/* |
| Disallow | /discussion/user-report-abuse/* |
| Disallow | /discussion/handlers/* |
| Disallow | /discussion/your-profile |
| Disallow | /discussion/your-comments |
| Disallow | /discussion/edit-profile |
| Disallow | /discussion/search/comments |
| Disallow | /discussion/* |
| Disallow | /search |
| Disallow | /music/artist/* |
| Disallow | /music/album/* |
| Disallow | /books/data/* |
| Disallow | /settings/ |
| Disallow | /embed/ |
| Disallow | /*styles/js-on.css$ |
| Disallow | /sport/olympics/2008/events/* |
| Disallow | /sport/olympics/2008/medals/* |
| Disallow | /f/healthcheck |
| Disallow | /sections |
| Disallow | /top-stories |
| Disallow | /most-read/sport |
| Disallow | /articles |
| Disallow | /global$ |
| Disallow | /*/feedarticle/* |
| Disallow | /travel/2013/aug/22/been-there-readers-competition?* |
| Disallow | /preference/* |
| Disallow | /59666047/ |
| Disallow | /print/ |
| Disallow | /info/tech-feedback |
| Disallow | /production-monitoring/ |
| Disallow | *.emailjson |
| Disallow | *.emailtxt |
| Disallow | /headline.txt |
| Disallow | *?*dcr=apps* |
Other Records
| Field | Value |
|---|---|
| sitemap | http://www.theguardian.com/sitemaps/news.xml |
| sitemap | http://www.theguardian.com/sitemaps/video.xml |
Warnings
- 2 invalid lines.
Comments