Log In Sign Up

austlii.edu.au
robots.txt

Robots Exclusion Standard data for austlii.edu.au

Resource Scan

Scan Details

Site Domain austlii.edu.au
Base Domain austlii.edu.au
Scan Status Ok
Last Scan2024-06-24T18:46:23+00:00
Next Scan 2024-07-24T18:46:23+00:00

Last Scan

Scanned2024-06-24T18:46:23+00:00
URL https://austlii.edu.au/robots.txt
Domain IPs 138.25.65.147
Response IP 138.25.65.147
Found Yes
Hash 532e5a056bfae900036bf861ffb259ba86d4d3d9198c3e318141624e9b565cb7
SimHash 3557bb7b7795

Groups

*

Rule Path
Disallow /au/cases/
Disallow /au/legis/cth/digest/
Disallow /au/other/
Disallow /austlii/stats/
Disallow /nz/cases/
Disallow /au/special/
Disallow /austlii/metstats/
Disallow /cases/
Disallow /cgi-bin/
Disallow /cgi-bin/sinodisp/
Disallow /cgi-dev/
Disallow /do/
Disallow /do2/
Disallow /LawCite/
Disallow /lawcite/
Disallow /incite/
Disallow /inCite/
Disallow /form/
Disallow /forms/
Disallow /fcgi-bin/
Disallow /rsjlibrary/rciadic/
Disallow /austlii/survey/
Disallow /*?
Disallow /~andrew/
Disallow /~armin/
Disallow /~jones/
Disallow /~joseph/
Disallow /~philip/
Disallow /~trev/
Disallow /~chris/

Other Records

Field Value
crawl-delay 120

gromit

Rule Path
Disallow /cgi-bin/
Disallow /fcgi-bin/
Disallow /do/
Disallow /do2/
Disallow /au/cases/
Disallow /au/legis/
Disallow /au/other/
Disallow /au/journals/
Disallow /au/special/
Disallow /nz/cases/

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /au/cases/
Disallow /au/legis/cth/digest/
Disallow /au/other/
Disallow /austlii/stats/
Disallow /nz/cases/
Disallow /au/special/
Disallow /austlii/metstats/

Other Records

Field Value
crawl-delay 120

australian business on-line

Rule Path
Disallow /au/legis/
Disallow /au/other/hca/
Disallow /au/other/iponline/
Disallow /au/other/ipaus/
Disallow /au/journals/
Disallow /rsjlibrary/rciadic
Disallow /cgi-bin/
Disallow /do/
Disallow /do2/
Disallow /lists/
Disallow /ombud/
Disallow /austlii/editors/
Disallow /austlii/FAQ
Disallow /form/
Disallow /fcgi-bin/

wang new zealand

Rule Path
Disallow /au/legis/
Disallow /au/cases/
Disallow /au/other/hca/
Disallow /au/other/iponline/
Disallow /au/other/ipaus/
Disallow /au/journals/
Disallow /rsjlibrary/rciadic
Disallow /cgi-bin/
Disallow /fcgi-bin/
Disallow /do/
Disallow /do2/
Disallow /lists/
Disallow /ombud/
Disallow /austlii/editors/
Disallow /austlii/FAQ
Disallow /nz/
Disallow /form/

googlebot

Rule Path
Allow /au/legis/
Allow /au/journals/
Allow /nz/journals/
Allow /cgi-bin/viewdb/au/legis/
Allow /cgi-bin/viewdb/au/journals/
Allow /cgi-bin/viewdb/nz/journals/
Allow /cgi-bin/viewdoc/au/legis/
Allow /cgi-bin/viewdoc/au/journals/
Allow /cgi-bin/viewdoc/nz/journals/
Disallow /au/cases
Disallow /nz/cases

slurp

Rule Path
Disallow /au/other/
Disallow /au/cases/
Disallow /au/legis/
Disallow /au/special/
Disallow /au/journals
Disallow /cgi-bin/
Disallow /fcgi-bin/
Disallow /form/
Disallow *.OLD/
Disallow *.NEW/
Disallow *.BUILD/
Disallow /au/journals/FedLawRw/2001/23.html
Back to top

Comments

  • Google adsbot ignores robots.txt unless specifically named!
  • User-agent: adsbot-google
  • Disallow: /
  • 2 July 2010 - unrestricted access to everything except the below
  • 5 Dec 2018
  • Added cases as somehow all cases indexed via this link that does not exist! TR 17 June 2011
  • Disallow indexing of dynamically generated pages eg search results
  • by Google
  • PTHC-20080107
  • Disallow access to personal directories
  • JO - 20160826
  • Australian Business On-Line
  • Created: PTHC-20000919
  • Wang -- you are flooding the server please e-mail us
  • User-agent: Wang New Zealand
  • Disallow: /
  • Googlebot --You flooding the server with request please e-mail us
  • 6 August 2002
  • Googlebot -- Allow google to index everything except cases
  • 14 August 2003
  • User-agent: Googlebot
  • Disallow: /au/cases
  • Disallow: /nz/cases
  • Googlebot -- Allow Google to index legis and journals
  • JO - 20230720
  • Slurp --You are flooding the server with request please e-mail us
  • 08 Oct 2002
  • User-agent: FAST Enterprise Crawler 6 used by LexisNexisAU
  • Fast crawler banned from crawling cases
  • Disallow: /au/cases/
  • Disallow: /au/other/
  • Disallow: /nz/cases/
  • Disallow: /cgi-bin/
  • Disallow: /cgi-dev/
  • Disallow: /do/
  • Disallow: /do2/
  • Disallow: /LawCite/
  • Disallow: /lawcite/
  • Disallow: /incite/
  • Disallow: /inCite/
  • Disallow: /form/
  • Disallow: /forms/
  • Disallow: /fcgi-bin/
  • Disallow: /rsjlibrary/rciadic
  • Disallow forbidden directories
  • Disallow forbidden files
Back to top