newsunlimited.co.uk
robots.txt
Robots Exclusion Standard data for newsunlimited.co.uk
Resource Scan
Scan Details
Site Domain | newsunlimited.co.uk |
Base Domain | newsunlimited.co.uk |
Scan Status | Failed |
Failure Reason | Scan timed out. |
Last Scan | 2024-05-08T20:06:25+00:00 |
Next Scan | 2024-07-07T20:06:25+00:00 |
Last Successful Scan
Scanned | 2021-10-16T15:59:58+00:00 |
URL | http://newsunlimited.co.uk/robots.txt |
Redirect | https://www.theguardian.com/robots.txt |
Redirect Domain | www.theguardian.com |
Redirect Base | theguardian.com |
Found | Yes |
Hash | 30d96e362ef58c65b94358afd43bda74ef8266733f6b33d2e5c06991baae543e |
SimHash | efa51fa1e7d4 |
Groups
*
Rule | Path |
---|---|
Disallow | /sendarticle/ |
Disallow | /Users/ |
Disallow | /users/ |
Disallow | /*/print$ |
Disallow | /email/ |
Disallow | /contactus/ |
Disallow | /share/ |
Disallow | /websearch |
Disallow | /*?commentpage= |
Disallow | /whsmiths/ |
Disallow | /external/overture/ |
Disallow | /discussion/report-abuse/* |
Disallow | /discussion/report-abuse-ajax/* |
Disallow | /discussion/comment-permalink/* |
Disallow | /discussion/report-abuse/* |
Disallow | /discussion/user-report-abuse/* |
Disallow | /discussion/handlers/* |
Disallow | /discussion/your-profile |
Disallow | /discussion/your-comments |
Disallow | /discussion/edit-profile |
Disallow | /discussion/search/comments |
Disallow | /discussion/* |
Disallow | /search |
Disallow | /music/artist/* |
Disallow | /music/album/* |
Disallow | /books/data/* |
Disallow | /settings/ |
Disallow | /embed/ |
Disallow | /*styles/js-on.css$ |
Disallow | /sport/olympics/2008/events/* |
Disallow | /sport/olympics/2008/medals/* |
Disallow | /f/healthcheck |
Disallow | /sections |
Disallow | /top-stories |
Disallow | /most-read/sport |
Disallow | /articles |
Disallow | /global$ |
Disallow | /*/feedarticle/* |
Disallow | /travel/2013/aug/22/been-there-readers-competition?* |
Disallow | /preference/* |
Disallow | /59666047/ |
Disallow | /print/ |
Disallow | /info/tech-feedback |
Disallow | /production-monitoring/ |
Disallow | *.emailjson |
Disallow | *.emailtxt |
Disallow | /headline.txt |
bingbot
Rule | Path |
---|---|
Disallow | /sendarticle/ |
Disallow | /Users/ |
Disallow | /users/ |
Disallow | /*/print$ |
Disallow | /email/ |
Disallow | /contactus/ |
Disallow | /share/ |
Disallow | /websearch |
Disallow | /*?commentpage= |
Disallow | /whsmiths/ |
Disallow | /external/overture/ |
Disallow | /discussion/report-abuse/* |
Disallow | /discussion/report-abuse-ajax/* |
Disallow | /discussion/comment-permalink/* |
Disallow | /discussion/report-abuse/* |
Disallow | /discussion/user-report-abuse/* |
Disallow | /discussion/handlers/* |
Disallow | /discussion/your-profile |
Disallow | /discussion/your-comments |
Disallow | /discussion/edit-profile |
Disallow | /discussion/search/comments |
Disallow | /discussion/* |
Disallow | /search |
Disallow | /music/artist/* |
Disallow | /music/album/* |
Disallow | /books/data/* |
Disallow | /settings/ |
Disallow | /embed/ |
Disallow | /*styles/js-on.css$ |
Disallow | /sport/olympics/2008/events/* |
Disallow | /sport/olympics/2008/medals/* |
Disallow | /f/healthcheck |
Disallow | /sections |
Disallow | /top-stories |
Disallow | /most-read/sport |
Disallow | /articles |
Disallow | /global$ |
Disallow | /*/feedarticle/* |
Disallow | /travel/2013/aug/22/been-there-readers-competition?* |
Disallow | /preference/* |
Disallow | /59666047/ |
Disallow | /print/ |
Disallow | /info/tech-feedback |
Disallow | /production-monitoring/ |
Disallow | *.emailjson |
Disallow | *.emailtxt |
Disallow | /headline.txt |
Other Records
Field | Value |
---|---|
crawl-delay | 1 |
Other Records
Field | Value |
---|---|
sitemap | http://www.theguardian.com/sitemaps/news.xml |
sitemap | http://www.theguardian.com/sitemaps/video.xml |
Comments