bigdug.co.uk
robots.txt
Robots Exclusion Standard data for bigdug.co.uk
Resource Scan
Scan Details
| Site Domain | bigdug.co.uk |
| Base Domain | bigdug.co.uk |
| Scan Status | Ok |
| Last Scan | 2024-11-04T21:04:18+00:00 |
| Next Scan | 2024-12-04T21:04:18+00:00 |
Last Scan
| Scanned | 2024-11-04T21:04:18+00:00 |
| URL | https://bigdug.co.uk/robots.txt |
| Redirect | https://www.bigdug.co.uk/robots.txt |
| Redirect Domain | www.bigdug.co.uk |
| Redirect Base | bigdug.co.uk |
| Domain IPs | 109.108.148.102 |
| Redirect IPs | 2600:9000:2024:1200:0:ee49:ae40:93a1, 2600:9000:2024:1a00:0:ee49:ae40:93a1, 2600:9000:2024:2600:0:ee49:ae40:93a1, 2600:9000:2024:3200:0:ee49:ae40:93a1, 2600:9000:2024:6000:0:ee49:ae40:93a1, 2600:9000:2024:7000:0:ee49:ae40:93a1, 2600:9000:2024:e200:0:ee49:ae40:93a1, 2600:9000:2024:fa00:0:ee49:ae40:93a1, 65.9.112.30, 65.9.112.68, 65.9.112.74, 65.9.112.80 |
| Response IP | 18.165.140.90 |
| Found | Yes |
| Hash | 8211506d828cf7fb7a8ef740868ac8272c34ffd74e69a83b54a2191703199605 |
| SimHash | 5c3bc775ee8e |
Groups
etaospider
baiduspider
brandverity
seekport crawler
blexbot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
*
| Rule | Path |
|---|---|
| Disallow | /ajax/removefrombasket/ |
| Disallow | /basket$ |
| Disallow | /checkout/ |
| Disallow | /errors/ |
| Disallow | /quickbuy/ |
| Disallow | /product_personalisation/ |
| Disallow | /register/ |
| Disallow | /vs/ |
| Disallow | /images/products/yofla/ |
| Disallow | /customer_groups/choose_group/ |
| Disallow | /formbuilder/ |
| Disallow | /callbacks/ |
| Disallow | /products/ |
| Disallow | /search/ |
| Disallow | /daily_deal/ |
| Disallow | /clearance-c18 |
| Disallow | /customer/login |
| Disallow | /testimonials/ |
| Disallow | /view-catalogue-i44 |
| Disallow | /secure-shopping-i52 |
| Disallow | /privacy-policy-i14 |
| Disallow | /faq-i45 |
| Disallow | /press/ |
| Disallow | /our-promise-i122 |
| Disallow | /quote-builder-i48 |
| Disallow | /productreviews/submit/ |
| Disallow | /payment-methods-i51 |
| Disallow | /first-time-buyer-i49 |
| Disallow | /request-a-call-back-i36 |
| Disallow | /cookies |
| Disallow | /upselling |
| Disallow | /ask-a-question-i24 |
| Disallow | /spring-deals-c1553 |
| Disallow | /reviews/ |
| Disallow | /productreviews/* |
| Disallow | /*?*sort= |
| Disallow | /*?*number-of-levels= |
| Disallow | /*?*colour= |
| Disallow | /*?*weight-loading-kg= |
| Disallow | /*?*height-mm= |
| Disallow | /*?*width-mm= |
| Disallow | /*?*depth-mm= |
| Disallow | /*?*litre-capacity= |
| Disallow | /*?*&*&* |
| Allow | /*utm*%26utm*%26utm |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.bigdug.co.uk/sitemap-index.xml |