Log In Sign Up

murdoch.edu.au
robots.txt

Robots Exclusion Standard data for murdoch.edu.au

Resource Scan

Scan Details

Site Domain murdoch.edu.au
Base Domain murdoch.edu.au
Scan Status Ok
Last Scan2024-05-14T10:08:44+00:00
Next Scan 2024-06-13T10:08:44+00:00

Last Scan

Scanned2024-05-14T10:08:44+00:00
URL https://www.murdoch.edu.au/robots.txt
Domain IPs 134.115.4.246
Response IP 134.115.4.246
Found Yes
Hash b2a438438466c72502cdb4b37de09f99c6b96cbfe296ef8291d5af8169ee0c71
SimHash b8968d8eed23

Groups

bad_robot

Rule Path
Disallow /test/apple/file1.html
Allow /cwisad
Allow /index/atoz
Disallow /

webcrawler
excite
omtrbot/1.0

Rule Path
Allow /
Disallow /test
Disallow /dev

*

Rule Path
Disallow /acad
Disallow /admin
Disallow /callista
Disallow /Ccpr
Disallow /cgi-bin/tt-check
Disallow /contacts
Disallow /copyright
Disallow /cwisad
Disallow /cwisindex
Disallow /cwistech
Disallow /dev
Disallow /directory
Disallow /dirs
Disallow /elaw
Disallow /feedback
Disallow /forms
Disallow /goto/stats
Disallow /help
Disallow /index
Disallow /index/feedback
Disallow /index/pageinfo
Disallow /index/policies/pageinfo
Disallow /its
Disallow /itservicedesk
Disallow /lbc
Disallow /oss
Disallow /oss2
Disallow /otherser
Disallow /portal
Disallow /search
Disallow /server/stats
Disallow /staff
Disallow /stats
Disallow /study/courses/course-structure/
Disallow /style
Disallow /synergy
Disallow /test
Disallow /vco
Disallow /wnew
Disallow /Careers-and-employment-centre/_document
Disallow /School-of-Veterinary-and-Life-Sciences/_document
Back to top

Comments

  • /robots.txt for http://www.murdoch.edu.au/
  • use <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
  • to suggest html files be skipped on a per file basis
  • Disallow: /Careers-and-employment-centre/_document/Newsletters/October-2015
  • Disallow: /Careers-and-employment-centre/_document/Newsletters/October-2015.pdf
Back to top