library.hud.ac.uk
robots.txt

Robots Exclusion Standard data for library.hud.ac.uk

Resource Scan

Scan Details

Site Domain library.hud.ac.uk
Base Domain hud.ac.uk
Scan Status Ok
Last Scan2025-07-20T18:41:10+00:00
Next Scan 2025-08-19T18:41:10+00:00

Last Scan

Scanned2025-07-20T18:41:10+00:00
URL https://library.hud.ac.uk/robots.txt
Domain IPs 20.49.155.210
Response IP 20.49.155.210
Found Yes
Hash 0d4c448c2240cf593e9aa929d3175fa8c4d916c8ed56cb93ede1a87d4f9b1175
SimHash 985cd944a8f1

Groups

semrushbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

quetextbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

*

Rule Path
Disallow /myreading/
Disallow /my/
Disallow /readinglists/
Disallow /archive/
Disallow /auth/
Disallow /perl/
Disallow /catlink/
Disallow /cgi/
Disallow /cas/
Disallow /ipac20/
Disallow /lemontree/
Disallow /login/
Disallow /rooms/
Disallow /user/
Disallow /calmview/
Disallow /CalmView/
Disallow /temp/
Disallow /files/temp/