bristol.ac.uk
robots.txt

Robots Exclusion Standard data for bristol.ac.uk

Resource Scan

Scan Details

Site Domain bristol.ac.uk
Base Domain bristol.ac.uk
Scan Status Ok
Last Scan2024-06-03T21:53:58+00:00
Next Scan 2024-07-03T21:53:58+00:00

Last Scan

Scanned2024-06-03T21:53:58+00:00
URL https://bristol.ac.uk/robots.txt
Domain IPs 137.222.1.237
Response IP 137.222.1.237
Found Yes
Hash aadfe95581ff853a5fe1a452b93008387ce468e05ac8f201d27c7e9e633d072b
SimHash c5394a75d715

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /comment
Disallow /contact/
Disallow /dev/
Disallow /esu/unitprogcat/
Disallow /eopro/
Disallow /feedback/
Disallow /fluff/
Disallow /imagelib/
Disallow /logs/
Disallow /news/news-results.html
Disallow /news/news-search.html?
Disallow /news/2001/clearing.html
Disallow /oenology
Disallow /prospectus/postgraduate/2013/
Disallow /prospectus/postgraduate/2014/
Disallow /prospectus/undergraduate/2014/
Disallow /slink/
Disallow /sso/login
Disallow /test/
Disallow /webcam/sport/archive/
Disallow /biology/people/person/
Disallow /composites/media/ICCM17/
Disallow /composites/media/icons/
Disallow /composites/media/icshm/
Disallow /Control_Panel/
Disallow /Depts/CAS/
Disallow /depts/CAS/
Disallow /geography/people/person/
Disallow /history/customs/resources/
Disallow /history/customs/datasets/
Disallow /is/cgi-bin/
Disallow /is/comment
Disallow /research/funding/salary_costing/
Disallow /study/postgraduate/search/research_interests
Disallow /study/postgraduate/search/search_results
Disallow /study/undergraduate/search/search_results
Disallow /media-library/sites/alspac/migrated/documents/
Disallow /it-services/advice/operatingsystems/mac/archive
Disallow /it-services/applications/collaboration/office/outlook/
Disallow /it-sysdev/out-of-service/
Disallow /it-sysdev/maintenance/
Disallow /*/comment$
Disallow /*/comment?url
Disallow /*/feedback$
Disallow /*/news-results.html
Disallow /*/news-search.html?
Disallow /*/safe/

claudebot

Rule Path
Disallow /unit-programme-catalogue

Comments

  • AFAIK only Googlebot understands these
  • Added by Dave, 2024-05-20 [Master Incident: 216543, DSI-2731: DDOS type traffic to UPC]