museumoflondon.org.uk
robots.txt
Robots Exclusion Standard data for museumoflondon.org.uk
Resource Scan
Scan Details
| Site Domain | museumoflondon.org.uk |
| Base Domain | museumoflondon.org.uk |
| Scan Status | Ok |
| Last Scan | 2024-05-01T21:01:49+00:00 |
| Next Scan | 2024-05-31T21:01:49+00:00 |
Last Scan
| Scanned | 2024-05-01T21:01:49+00:00 |
| URL | https://museumoflondon.org.uk/robots.txt |
| Redirect | https://www.museumoflondon.org.uk/robots.txt |
| Redirect Domain | www.museumoflondon.org.uk |
| Redirect Base | museumoflondon.org.uk |
| Domain IPs | 104.22.30.77, 104.22.31.77, 172.67.14.208, 2606:4700:10::6816:1e4d, 2606:4700:10::6816:1f4d, 2606:4700:10::ac43:ed0 |
| Redirect IPs | 104.22.30.77, 104.22.31.77, 172.67.14.208, 2606:4700:10::6816:1e4d, 2606:4700:10::6816:1f4d, 2606:4700:10::ac43:ed0 |
| Response IP | 172.67.14.208 |
| Found | Yes |
| Hash | 76af83baec18e773cdc9ac05cabbbc31ac993c41476cba4f21fd23280508b4d7 |
| SimHash | 6e0814c8c095 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /application/attributes |
| Disallow | /application/authentication |
| Disallow | /application/bootstrap |
| Disallow | /application/config |
| Disallow | /application/controllers |
| Disallow | /application/elements |
| Disallow | /application/helpers |
| Disallow | /application/jobs |
| Disallow | /application/languages |
| Disallow | /application/mail |
| Disallow | /application/models |
| Disallow | /application/page_types |
| Disallow | /application/single_pages |
| Disallow | /application/tools |
| Disallow | /application/views |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |