gaborshoes.co.uk
robots.txt
Robots Exclusion Standard data for gaborshoes.co.uk
Resource Scan
Scan Details
| Site Domain | gaborshoes.co.uk |
| Base Domain | gaborshoes.co.uk |
| Scan Status | Ok |
| Last Scan | 2024-10-24T16:46:59+00:00 |
| Next Scan | 2024-11-23T16:46:59+00:00 |
Last Scan
| Scanned | 2024-10-24T16:46:59+00:00 |
| URL | https://gaborshoes.co.uk/robots.txt |
| Redirect | https://www.gaborshoes.co.uk/robots.txt |
| Redirect Domain | www.gaborshoes.co.uk |
| Redirect Base | gaborshoes.co.uk |
| Domain IPs | 109.108.148.102 |
| Redirect IPs | 80.66.203.177 |
| Response IP | 80.66.203.177 |
| Found | Yes |
| Hash | 0b82d9b88d31d9429856903f7ea0c6f8230dba3e29bfde40747f20533238b16d |
| SimHash | 760fd770ce1a |
Groups
etaospider
baiduspider
brandverity
seekport crawler
seekportbot
blexbot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
| crawl-delay | 60 |
| crawl-delay | 30 |
charlotte
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
| crawl-delay | 1 |
seznambot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /ajax/removefrombasket/ |
| Disallow | /basket$ |
| Disallow | /checkout/ |
| Disallow | /errors/ |
| Disallow | /quickbuy/ |
| Disallow | /product_personalisation/ |
| Disallow | /register/ |
| Disallow | /vs/ |
| Disallow | /images/products/yofla/ |
| Disallow | /customer_groups/choose_group/ |
| Disallow | /formbuilder/ |
| Disallow | /callbacks/ |
| Disallow | /ajax/formatprice/ |
| Disallow | /products/ |
| Disallow | /productimages/gallery/ |
| Disallow | /search |
| Disallow | /wishlist/ |
| Disallow | /basket |
| Disallow | /checkout |
| Disallow | /compare |
| Disallow | *?project=* |
| Disallow | /map/?branch_select |
| Disallow | /*?sort* |
| Disallow | *?view_all* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.gaborshoes.co.uk/sitemap-index.xml |
Warnings
- `request-rate` is not a known field.