Log In Sign Up

bu.edu
robots.txt

Robots Exclusion Standard data for bu.edu

Resource Scan

Scan Details

Site Domain bu.edu
Base Domain bu.edu
Scan Status Ok
Last Scan2024-10-20T21:00:22+00:00
Next Scan 2024-11-19T21:00:22+00:00

Last Scan

Scanned2024-10-20T21:00:22+00:00
URL https://bu.edu/robots.txt
Redirect https://www.bu.edu/robots.txt
Redirect Domain www.bu.edu
Redirect Base bu.edu
Domain IPs 128.197.236.4
Redirect IPs 54.230.112.122, 54.230.112.30, 54.230.112.88, 54.230.112.98
Response IP 18.165.140.59
Found Yes
Hash 257fb67183e56ba650cb8dc8528058d4e60603c6412357115877bd7c0f6e7aba
SimHash 62bfd353af50

Groups

linklint

Rule Path
Disallow /workaroundForLinkLintRandomDirForConfig/

w3c-checklink

Rule Path
Disallow /cms/
Disallow /cgi-bin/
Disallow /htbin/
Disallow /htbin.ph/
Disallow /BUbin/
Disallow /bubin/
Disallow /testing/
Disallow /TESTING/
Disallow /IT/SoftwareDist/
Disallow /it/SoftwareDist/
Disallow /software/
Disallow /SOFTWARE/
Disallow /IT/new/
Disallow /it/new/
Disallow /nis/
Disallow /nishd/
Disallow /library/working/
Disallow /library/WORKING/
Disallow /reports/
Disallow /bulletins/work/
Disallow /admissions/test/
Disallow /cas/oldsite/
Disallow /MPA/
Disallow /finaid/test/
Disallow /naitest/
Disallow /newswire/
Disallow /practice/
Disallow /providers/
Disallow /stats/
Disallow /usc/test/
Disallow /webcentral/output/
Disallow /webmail/
Disallow /alumni/portfolio/
Disallow /dev/

gsa-crawler

Rule Path
Disallow /cgi-bin/
Disallow /cms/
Disallow /htbin/
Disallow /htbin.ph/
Disallow /BUbin/
Disallow /bubin/
Disallow /testing/
Disallow /TESTING/
Disallow /IT/SoftwareDist/
Disallow /it/SoftwareDist/
Disallow /software/
Disallow /SOFTWARE/
Disallow /IT/new/
Disallow /it/new/
Disallow /library/working/
Disallow /library/WORKING/
Disallow /reports/
Disallow /nisdev/
Disallow /bulletins/work/
Disallow /admissions/test/
Disallow /cas/oldsite/
Disallow /MPA/
Disallow /finaid/test/
Disallow /naitest/
Disallow /newswire/
Disallow /practice/
Disallow /providers/
Disallow /stats/
Disallow /usc/test/
Disallow /webcentral/output/
Disallow /webmail/
Disallow /alumni/portfolio/
Disallow /dbin/dos/ocs/
Disallow /dev/
Disallow /wbur/arts/
Disallow /wbur/connection/
Disallow /wbur/herenow/
Disallow /wbur/livingonearth/
Disallow /wbur/miscellaneous/
Disallow /wbur/onpoint/
Disallow /wbur/special_projects_unit/
Disallow /wbur/wburnews/
Disallow /wbur/woi/
Disallow /link/
Disallow /home-media/
Disallow /buniverse/add/
Disallow /buniverse/admin/
Disallow /buniverse/buniverse1/
Disallow /buniverse/contact/
Disallow /buniverse/cron/
Disallow /buniverse/data/
Disallow /buniverse/delete/
Disallow /buniverse/edit/
Disallow /buniverse/embed/
Disallow /buniverse/login/
Disallow /buniverse/logout/
Disallow /buniverse/messages/
Disallow /buniverse/my-videos/
Disallow /buniverse/search/
Disallow /buniverse/support/
Disallow /buniverse/util/
Disallow /buniverse/viewed/
Disallow /buniverse/vote/
Disallow /buniverse/youtube/
Disallow /summer/archive/

netsparker

Rule Path
Disallow /

*

Rule Path
Disallow /cms/
Disallow /cgi-bin/
Disallow /htbin/
Disallow /htbin.ph/
Disallow /BUbin/
Disallow /bubin/
Disallow /testing/
Disallow /TESTING/
Disallow /IT/SoftwareDist/
Disallow /it/SoftwareDist/
Disallow /software/
Disallow /SOFTWARE/
Disallow /IT/new/
Disallow /it/new/
Disallow /nis/
Disallow /library/working/
Disallow /library/WORKING/
Disallow /reports/
Disallow /nisdev/
Disallow /bulletins/work/
Disallow /admissions/test/
Disallow /cas/oldsite/
Disallow /MPA/
Disallow /finaid/test/
Disallow /naitest/
Disallow /newswire/
Disallow /practice/
Disallow /providers/
Disallow /stats/
Disallow /usc/test/
Disallow /webcentral/output/
Disallow /webmail/
Disallow /alumni/portfolio/
Disallow /dbin/dos/ocs/
Disallow /dev/
Disallow /wbur/arts/
Disallow /wbur/connection/
Disallow /wbur/herenow/
Disallow /wbur/livingonearth/
Disallow /wbur/miscellaneous/
Disallow /wbur/onpoint/
Disallow /wbur/special_projects_unit/
Disallow /wbur/wburnews/
Disallow /wbur/woi/
Disallow /link/
Disallow /home-media/
Disallow /buniverse/add/
Disallow /buniverse/admin/
Disallow /buniverse/buniverse1/
Disallow /buniverse/contact/
Disallow /buniverse/cron/
Disallow /buniverse/data/
Disallow /buniverse/delete/
Disallow /buniverse/edit/
Disallow /buniverse/embed/
Disallow /buniverse/login/
Disallow /buniverse/logout/
Disallow /buniverse/messages/
Disallow /buniverse/my-videos/
Disallow /buniverse/search/
Disallow /buniverse/support/
Disallow /buniverse/util/
Disallow /buniverse/viewed/
Disallow /buniverse/vote/
Disallow /buniverse/youtube/
Disallow /academics/archive/
Disallow /summer/archive/

Other Records

Field Value
crawl-delay 15
Back to top

Comments

  • Directions for robots. See this URL:
  • http://info.webcrawler.com/mak/projects/robots/norobots.html
  • for a description of the file format.
  • 2008-08-21
  • Here is where we override the default action
  • Due to a bug in linklint, must first specify a disallow in order for
  • for all other directories to be allowed. Feel free to add other
  • disallows below the first disallow line.
  • Allow W3C link Validator for /dev/ and /nisdev/
  • skipping other dynamic content or private areas
  • 2004-08-27 gaudette
  • default action - currently it allows access to most of the site
  • skipping dynamic content or private areas
  • BUniverse exclusions added by kgrin on 2010-04-26
  • Emergency change 2012-02-14 bfenster, in response to incident
  • Emergency change 2014-11-17 bfenster, in response to incident
  • default action - currently it allows access to most of the site
  • skipping dynamic content or private areas
  • BUniverse exclusions added by kgrin on 2010-04-21
  • academics/summer archive exclusions added by kgrin on 2011-07-17
Back to top

Warnings

  • 2 invalid lines.
Back to top