conted.ox.ac.uk
robots.txt
Robots Exclusion Standard data for conted.ox.ac.uk
Resource Scan
Scan Details
| Site Domain | conted.ox.ac.uk |
| Base Domain | ox.ac.uk |
| Scan Status | Ok |
| Last Scan | 2024-11-18T17:12:27+00:00 |
| Next Scan | 2024-12-18T17:12:27+00:00 |
Last Scan
| Scanned | 2024-11-18T17:12:27+00:00 |
| URL | https://conted.ox.ac.uk/robots.txt |
| Domain IPs | 129.67.165.148 |
| Response IP | 129.67.165.148 |
| Found | Yes |
| Hash | b7fd32222c35a552946b8514cb8711d29d231163ea30e37334ecfc4e920dd490 |
| SimHash | 4063695dc7d1 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | Block pdf files. Non-standard but works for major search engines. | |
| Disallow | /basket/add-module*$ | - |
| Disallow | /error$ | - |
| Disallow | /events/log-payment/*$ | Block any log payment events functions to be crawled. |
| Disallow | /events/register/*$ | - |
| Disallow | /about-us | - |
| Disallow | /about/coronavirus-faq | - |
| Disallow | /coronavirus-faq | - |
| Disallow | /about/continuing-education-library-service-during-closure | - |
| Disallow | /mnt/archive/*$ | - |
| Disallow | /mnt/attachments/*$ | - |
| Disallow | /mnt/audio/*$ | - |
| Disallow | /mnt/forms/*$ | - |
Warnings
- 1 invalid line.
Comments