Log In Sign Up

students.mq.edu.au
robots.txt

Robots Exclusion Standard data for students.mq.edu.au

Resource Scan

Scan Details

Site Domain students.mq.edu.au
Base Domain mq.edu.au
Scan Status Ok
Last Scan2025-08-31T01:23:20+00:00
Next Scan 2025-09-30T01:23:20+00:00

Last Scan

Scanned2025-08-31T01:23:20+00:00
URL https://students.mq.edu.au/robots.txt
Domain IPs 43.245.41.37
Response IP 43.245.41.37
Found Yes
Hash 71496fa4884ba95f116fc7fbfd4dfb57ef9db3ac6c320335c8cc1efce45a5f94
SimHash 2d01781287a2

Groups

funnelback

Rule Path
Allow /
Disallow /uat/*

bingbot

Rule Path
Disallow /search
Disallow /pubstatic/*
Disallow /mq_templates/*
Disallow /study/new-ia*
Disallow /*?sq_content_src=
Disallow /*_recache
Disallow /*_edit
Disallow /*_admin
Disallow /*_login
Disallow /*_performance
Disallow /*_design
Disallow /*_web_services
Disallow /*_guide
Disallow /*_integration
Disallow /uat/*

Other Records

Field Value
crawl-delay 2

*

Rule Path
Allow /
Disallow /*?sq_content_src=
Disallow /*_recache
Disallow /*_edit
Disallow /*_admin
Disallow /*_login
Disallow /*_performance
Disallow /*_design
Disallow /*_web_services
Disallow /*_guide
Disallow /*_integration
Disallow /uat/*
Disallow /search

googlebot

Rule Path
Disallow /search
Back to top

Other Records

Field Value
sitemap https://students.mq.edu.au/sitemap.xml
Back to top

Comments

  • Limit Bingbot
  • Allow CSS and JS
  • Allow: /_design/css/*
  • Allow: /_design/js/*
  • Disallow some matrix defaults
  • Disallow UAT pages
  • Disallow search page
  • Sitemap
  • Keep deliberately duplicated pages out of the Google index
Back to top

Warnings

  • `noindex` is not a known field.
Back to top