whsmith.co.uk
robots.txt
Robots Exclusion Standard data for whsmith.co.uk
Resource Scan
Scan Details
| Site Domain | whsmith.co.uk |
| Base Domain | whsmith.co.uk |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-11-09T19:23:19+00:00 |
| Next Scan | 2025-01-08T19:23:19+00:00 |
Last Successful Scan
| Scanned | 2024-08-19T16:54:36+00:00 |
| URL | https://www.whsmith.co.uk/robots.txt |
| Domain IPs | 13.227.254.128, 13.227.254.24, 13.227.254.25, 13.227.254.33 |
| Response IP | 13.227.254.128 |
| Found | Yes |
| Hash | 8ef0d2f2539b2f374196da820914b1fd9fdd7b222a0a0747de0d852968f8388b |
| SimHash | 44b67240f123 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /on/demandware.store/ |
| Disallow | */s/whsmith/dw/ |
| Disallow | /register$ |
| Disallow | /register/ |
| Disallow | /login$ |
| Disallow | /login/ |
| Disallow | /my-profile$ |
| Disallow | /my-profile/ |
| Disallow | /account$ |
| Disallow | /account/ |
| Disallow | /account-editpassword |
| Disallow | /forgotten-password |
| Disallow | /address-book/ |
| Disallow | /address-book$ |
| Disallow | /shopping-basket |
| Disallow | /shopping-basket/ |
| Disallow | /checkout-login |
| Disallow | /checkout-shipping |
| Disallow | /checkout-payment |
| Disallow | /checkout-confirm |
| Disallow | /orders$ |
| Disallow | /orders? |
| Disallow | /orders/ |
| Disallow | /search/ |
| Disallow | /search?q= |
| Disallow | *search?keywords= |
| Disallow | */search?cgid |
| Disallow | /?q= |
| Disallow | *?c_ |
| Disallow | *?brand= |
| Disallow | *?price= |
| Disallow | *?srule= |
| Disallow | *?view= |
| Disallow | */product-image/zoom/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.whsmith.co.uk/sitemap_index.xml |
Comments