Log In Sign Up

exeter.ac.uk
robots.txt

Robots Exclusion Standard data for exeter.ac.uk

Resource Scan

Scan Details

Site Domain exeter.ac.uk
Base Domain exeter.ac.uk
Scan Status Ok
Last Scan2024-04-29T06:43:44+00:00
Next Scan 2024-05-29T06:43:44+00:00

Last Scan

Scanned2024-04-29T06:43:44+00:00
URL https://exeter.ac.uk/robots.txt
Redirect https://www.exeter.ac.uk/robots.txt
Redirect Domain www.exeter.ac.uk
Redirect Base exeter.ac.uk
Domain IPs 3.248.175.28, 34.254.218.29, 52.49.158.100
Redirect IPs 13.33.88.125, 13.33.88.46, 13.33.88.86, 13.33.88.91, 2600:9000:223b:1600:1d:8ab5:b1c0:93a1, 2600:9000:223b:200:1d:8ab5:b1c0:93a1, 2600:9000:223b:4a00:1d:8ab5:b1c0:93a1, 2600:9000:223b:6400:1d:8ab5:b1c0:93a1, 2600:9000:223b:a200:1d:8ab5:b1c0:93a1, 2600:9000:223b:b200:1d:8ab5:b1c0:93a1, 2600:9000:223b:ca00:1d:8ab5:b1c0:93a1, 2600:9000:223b:e800:1d:8ab5:b1c0:93a1
Response IP 13.33.88.46
Found Yes
Hash 808b63f028fca34471eda64783509c111f3f99afd422cdaec43fa92006611b99
SimHash 296d1343b95e

Groups

htdig

Product Comment
htdig ht://Dig should avoid
Rule Path Comment
Disallow /cgi-bin/ Script files
Disallow /errordocs/ Local Error docs
Disallow /spool/ Local MOMspider output
Disallow /web/ Local Web Tools
Disallow /config/ Local Web Config
Disallow /cgi-src/ Local CGI Source
Disallow /brad/noindex/ Brads noindex text stuff
Disallow /~BBagilho/noindex/ Brads noindex text stuff
Disallow /alumni/ Local alumni
Disallow /alumnidb/ Local alumni
Disallow /iais/pgc/ Causes looping
Disallow /its/mwr/ Web Stats
Disallow /its/meetingmaker/ MeetingMaker
Disallow /its/software/cd/localstuff/ ITS CD info
Disallow /its/email2003/ email2003
Disallow /tusage/ Web Stats
Disallow /library/ Library web pages have moved
Disallow /wusage/ Web Stats
Disallow /web/ Common Manuals, etc
Disallow /projects/admin/ Personnel files
Disallow /Projects/admin/ Personnel files

*

Product Comment
* All other spiders should avoid
Rule Path Comment
Disallow /cgi-bin/ Script files
Disallow /errordocs/ Local Error docs
Disallow /spool/ Local MOMspider output
Disallow /web/ Local Web Tools
Disallow /config/ Local Web Config
Disallow /cgi-src/ Local CGI Source
Disallow /noindex/ noindex stuff
Disallow /no-index/ noindex stuff
Disallow /alumni/ Local alumni
Disallow /alumnidb/ Local alumni
Disallow /alumnisupporters/contactus/updateyourdetails/ Local alumni
Disallow /media/universityofexeter/alumniandsupporters/givingtoexeter/disbursement2013/ -
Disallow /its/mwr/ Web Stats
Disallow /its/meetingmaker/ MeetingMaker
Disallow /its/software/cd/localstuff/ ITS CD info
Disallow /its/email2003/ email2003
Disallow /its/ipc2003/ ipc2003
Disallow /tusage/ Web Stats
Disallow /wusage/ Web Stats
Disallow /web/ Common Manuals, etc
Disallow /proxy/ Local proxy info
Disallow /admin/personnel/ Personnel files cause looping process
Disallow /projects/admin/ Personnel files
Disallow /Projects/admin/ Personnel files
Disallow /about/governance/business_areas/fundraising/index.shtml -
Disallow /departments/communication/marketing/aboutus/ -
Disallow /undergraduate/2010/ Archived files
Disallow /undergraduate/2011/ Archived files
Disallow /undergraduate/2012/ Archived files
Disallow /undergraduate/2013/ Archived files
Disallow /undergraduate/2014/ Archived files
Disallow /undergraduate/2015/ Archived files
Disallow /undergraduate/2016/ Archived files
Disallow /undergraduate/2017/ Archived files
Disallow /undergraduate/2018/ Archived files
Disallow /undergraduate/2019/ Archived files
Disallow /undergraduate/degrees2014/ Archived files
Disallow /undergraduate/courses-by-subject-2014/ Archived files
Disallow /undergraduate/all-courses-2014/ Archived files
Disallow /undergraduate/degrees2015/ Archived files
Disallow /undergraduate/courses-by-subject-2015/ Archived files
Disallow /undergraduate/all-courses-2015/ Archived files
Disallow /undergraduate/degrees2016/ Archived files
Disallow /undergraduate/courses-by-subject-2016/ Archived files
Disallow /undergraduate/all-courses-2016/ Archived files
Disallow /undergraduate/degrees2017/ Archived files
Disallow /undergraduate/courses-by-subject-2017/ Archived files
Disallow /undergraduate/all-courses-2017/ Archived files
Disallow /undergraduate/degrees2018/ Archived files
Disallow /undergraduate/courses-by-subject-2018/ Archived files
Disallow /undergraduate/all-courses-2018/ Archived files
Disallow /undergraduate/degrees2019/ Archived files
Disallow /undergraduate/courses-by-subject-2019/ Archived files
Disallow /undergraduate/all-courses-2019/ Archived files
Disallow /undergraduate/courses2020/ Archived files
Disallow /undergraduate/courses2021/ Archived files
Disallow /postgraduate/pgp2011/ Archived files
Disallow /postgraduate/pgp2012/ Archived files
Disallow /postgraduate/pgp2013/ Archived files
Disallow /postgraduate/pgp2014/ Archived files
Disallow /postgraduate/pgp2015/ Archived files
Disallow /postgraduate/pgp2016/ Archived files
Disallow /postgraduate/pgp2018/ Archived files
Disallow /postgraduate/pgp2019/ Archived files
Disallow /codebox/ Archived files
Disallow /staff/policies/calendar/archive/ Calendar archive
Disallow /staff/psr/ -
Disallow /cornwall/news/articles/index.html Hidden responsive news pages
Disallow /business/news/articles/index.html Hidden responsive news pages
Disallow /students/news/articles/index.html Hidden responsive news pages
Disallow /research/marine/news/articles/index.html Hidden responsive news pages
Disallow /teaching-excellence/news/articles/index.html Hidden responsive news pages
Disallow /campusservices/news/articles/index.html Hidden responsive news pages
Disallow /study-dev/ Development site

googlebot

Rule Path Comment
Disallow /undergraduate/2010/ Archived files
Disallow /undergraduate/2011/ Archived files
Disallow /undergraduate/2012/ Archived files
Disallow /undergraduate/2013/ Archived files
Disallow /undergraduate/2014/ Archived files
Disallow /undergraduate/2015/ Archived files
Disallow /undergraduate/2016/ Archived files
Disallow /undergraduate/2017/ Archived files
Disallow /undergraduate/2018/ Archived files
Disallow /undergraduate/2019/ Archived files
Disallow /undergraduate/degrees2014/ Archived files
Disallow /undergraduate/courses-by-subject-2014/ Archived files
Disallow /undergraduate/all-courses-2014/ Archived files
Disallow /undergraduate/degrees2015/ Archived files
Disallow /undergraduate/courses-by-subject-2015/ Archived files
Disallow /undergraduate/all-courses-2015/ Archived files
Disallow /undergraduate/degrees2016/ Archived files
Disallow /undergraduate/courses-by-subject-2016/ Archived files
Disallow /undergraduate/all-courses-2016/ Archived files
Disallow /undergraduate/2016/ Archived files
Disallow /undergraduate/degrees2017/ Archived files
Disallow /undergraduate/courses-by-subject-2017/ Archived files
Disallow /undergraduate/all-courses-2017/ Archived files
Disallow /undergraduate/degrees2018/ Archived files
Disallow /undergraduate/courses-by-subject-2018/ Archived files
Disallow /undergraduate/all-courses-2018/ Archived files
Disallow /undergraduate/degrees2019/ Archived files
Disallow /undergraduate/courses-by-subject-2019/ Archived files
Disallow /undergraduate/all-courses-2019/ Archived files
Disallow /undergraduate/courses2020/ Archived files
Disallow /undergraduate/courses2021/ Archived files
Disallow /postgraduate/pgp2011/ Archived files
Disallow /postgraduate/pgp2012/ Archived files
Disallow /postgraduate/pgp2013/ Archived files
Disallow /postgraduate/pgp2014/ Archived files
Disallow /postgraduate/pgp2015/ Archived files
Disallow /postgraduate/pgp2016/ Archived files
Disallow /postgraduate/pgp2018/ Archived files
Disallow /postgraduate/pgp2019/ Archived files
Disallow /cornwall/news/articles/index.html Hidden responsive news pages
Disallow /business/news/articles/index.html Hidden responsive news pages
Disallow /students/news/articles/index.html Hidden responsive news pages
Disallow /research/marine/news/articles/index.html Hidden responsive news pages
Disallow /teaching-excellence/news/articles/index.html Hidden responsive news pages
Disallow /study-dev/ Development site
Disallow /staff/policies/calendar/archive/ Calendar archive
Disallow /study/undergraduate/brochures/2023/ UG prospectus
Back to top

Other Records

Field Value
sitemap https://www.exeter.ac.uk/sitemap.xml
Back to top

Comments

  • $Id: robots.txt,v 1.2 1996/06/26 04:15:40 W.Edmunds Exp $
  • robots.txt for http://www.ex.ac.uk/
  • see <http://web.nexor.co.uk/mak/doc/robots/norobots.html> for an explanation.
  • Disallow: /admin/personnel/ # Personnel files cause looping process
Back to top