Log In Sign Up

courses.maths.ox.ac.uk
robots.txt

Robots Exclusion Standard data for courses.maths.ox.ac.uk

Resource Scan

Scan Details

Site Domain courses.maths.ox.ac.uk
Base Domain ox.ac.uk
Scan Status Ok
Last Scan2025-11-30T11:24:13+00:00
Next Scan 2025-12-14T11:24:13+00:00

Last Scan

Scanned2025-11-30T11:24:13+00:00
URL https://courses.maths.ox.ac.uk/robots.txt
Domain IPs 129.67.184.134, 2001:630:441:201::8143:b886
Response IP 129.67.184.134
Found Yes
Hash ced5ef1804d50b1aa2f923943e6c3461b883aa9a2505390cf411e5c4d49c776a
SimHash 7406734408b5

Groups

*

Rule Path
Disallow /pluginfile.php/

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • Delay included for SiteSucker - https://ricks-apps.com/osx/sitesucker/archive/5.x/5.1.x/5.1/manuals/en/pgs/Request.html
  • Discourage robots from downloading large files
Back to top