5pm.co.uk
robots.txt
Robots Exclusion Standard data for 5pm.co.uk
Resource Scan
Scan Details
| Site Domain | 5pm.co.uk |
| Base Domain | 5pm.co.uk |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-03-17T17:27:51+00:00 |
| Next Scan | 2025-06-15T17:27:51+00:00 |
Last Successful Scan
| Scanned | 2024-08-26T13:07:13+00:00 |
| URL | https://www.5pm.co.uk/robots.txt |
| Redirect | https://koobleit.com/robots.txt |
| Redirect Domain | koobleit.com |
| Redirect Base | koobleit.com |
| Domain IPs | 172.66.40.111, 172.66.43.145, 2606:4700:3108::ac42:286f, 2606:4700:3108::ac42:2b91 |
| Redirect IPs | 15.197.167.90, 3.33.186.135 |
| Response IP | 3.33.186.135 |
| Found | Yes |
| Hash | 1104fd2819760419f8740d85bb34d34da3ee75f984b078af83caf242bcf283a1 |
| SimHash | 470c5c4107f3 |
Groups
amazonbot
| Rule | Path |
|---|---|
| Disallow | / |
| Disallow | /the-big-deal/booking/ |
| Disallow | /takeaway/booking/ |
| Disallow | /health-beauty/search/ |
| Disallow | /restaurant/search/ |
| Disallow | /health-beauty/booking/ |
| Disallow | /restaurant/booking/ |
bingbot
| Rule | Path |
|---|---|
| Disallow | /the-big-deal/booking/ |
| Disallow | /takeaway/booking/ |
| Disallow | /health-beauty/search/all/ |
| Disallow | /health-beauty/booking/ |
| Disallow | /restaurant/booking/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /the-big-deal/booking/ |
| Disallow | /takeaway/booking/ |
| Disallow | /health-beauty/search/all/ |
| Disallow | /health-beauty/booking/ |
| Disallow | /restaurant/booking/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
rogerbot
| Rule | Path |
|---|---|
| Disallow | /the-big-deal/booking/ |
| Disallow | /takeaway/booking/ |
| Disallow | /health-beauty/search/all/ |
| Disallow | /health-beauty/booking/ |
| Disallow | /restaurant/booking/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
*
| Rule | Path |
|---|---|
| Disallow | /the-big-deal/booking/ |
| Disallow | /takeaway/booking/ |
| Disallow | /health-beauty/search/all/ |
| Disallow | /health-beauty/booking/ |
| Disallow | /restaurant/booking/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.koobleit.com/sitemap.xml |
Comments