Log In Sign Up

mcgill.ca
robots.txt

Robots Exclusion Standard data for mcgill.ca

Resource Scan

Scan Details

Site Domain mcgill.ca
Base Domain mcgill.ca
Scan Status Ok
Last Scan2024-11-15T10:43:45+00:00
Next Scan 2024-12-15T10:43:45+00:00

Last Scan

Scanned2024-11-15T10:43:45+00:00
URL https://mcgill.ca/robots.txt
Domain IPs 132.216.177.160
Response IP 132.216.177.160
Found Yes
Hash 0ccc56fcea80253e9ac631776a059fcd52ac49e92d72fc0b9b3428e41f3eab3f
SimHash a4903d19c574

Groups

lucidworks-anda/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

*

Rule Path
Allow */misc/*.css$
Allow */misc/*.css?
Allow */misc/*.js$
Allow */misc/*.js?
Allow */misc/*.gif
Allow */misc/*.jpg
Allow */misc/*.jpeg
Allow */misc/*.png
Allow */modules/*.css$
Allow */modules/*.css?
Allow */modules/*.js$
Allow */modules/*.js?
Allow */modules/*.gif
Allow */modules/*.jpg
Allow */modules/*.jpeg
Allow */modules/*.png
Allow */profiles/*.css$
Allow */profiles/*.css?
Allow */profiles/*.js$
Allow */profiles/*.js?
Allow */profiles/*.gif
Allow */profiles/*.jpg
Allow */profiles/*.jpeg
Allow */profiles/*.png
Allow */themes/*.css$
Allow */themes/*.css?
Allow */themes/*.js$
Allow */themes/*.js?
Allow */themes/*.gif
Allow */themes/*.jpg
Allow */themes/*.jpeg
Allow */themes/*.png
Allow */study/2024-2025
Disallow */includes/
Disallow */modules/
Disallow */profiles/
Disallow */scripts/
Disallow */themes/
Disallow */study/2009-2010
Disallow */study/2010-2011
Disallow */study/2011-2012
Disallow */study/2012-2013
Disallow */study/2013-2014
Disallow */study/2014-2015
Disallow */study/2015-2016
Disallow */study/2016-2017
Disallow */study/2017-2018
Disallow */study/2018-2019
Disallow */study/2019-2020
Disallow */study/2020-2021
Disallow */study/2021-2022
Disallow */study/2022-2023
Disallow */study/2023-2024
Disallow */CHANGELOG.txt
Disallow */cron.php
Disallow */INSTALL.mysql.txt
Disallow */INSTALL.pgsql.txt
Disallow */INSTALL.sqlite.txt
Disallow */install.php
Disallow */INSTALL.txt
Disallow */LICENSE.txt
Disallow */MAINTAINERS.txt
Disallow */update.php
Disallow */UPGRADE.txt
Disallow */xmlrpc.php
Disallow */misc/favicon.ico
Disallow */admin/
Disallow */comment/reply/
Disallow */filter/tips/
Disallow */node/add/
Disallow */search/
Disallow /study/*/courses/search
Disallow /study/*/programs/search
Disallow /study/*/search/all
Disallow */user/register/
Disallow */user/password/
Disallow */user/login/
Disallow */user/logout/
Disallow */user
Disallow */?q=admin%2F
Disallow */?q=comment%2Freply%2F
Disallow */?q=filter%2Ftips%2F
Disallow */?q=node%2Fadd%2F
Disallow */?q=search%2F
Disallow /?q=study%2F*%2Fcourses%2Fsearch
Disallow /?q=study%2F*%2Fprograms%2Fsearch
Disallow /?q=study%2F*%2Fsearch%2Fall
Disallow */?q=user%2Fpassword%2F
Disallow */?q=user%2Fregister%2F
Disallow */?q=user%2Flogin%2F
Disallow */?q=user%2Flogout%2F

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
Back to top