Log In Sign Up

spu.edu
robots.txt

Robots Exclusion Standard data for spu.edu

Resource Scan

Scan Details

Site Domain spu.edu
Base Domain spu.edu
Scan Status Ok
Last Scan2024-11-17T04:26:23+00:00
Next Scan 2024-12-17T04:26:23+00:00

Last Scan

Scanned2024-11-17T04:26:23+00:00
URL https://spu.edu/robots.txt
Domain IPs 20.120.185.245
Response IP 20.120.185.245
Found Yes
Hash 5e69e56a9fca29869bf1c68ce090f075dd306d0c4ca46d99fa40fc023353826a
SimHash bbc02dfd4cd3

Groups

*

Rule Path
Disallow /facstaff/
Disallow /classifieds
Disallow /classifieds/
Disallow /Classifieds/
Disallow /CLASSIFIEDS/
Disallow /webapps/staff_directory/
Disallow /administration/events/downtown-business-breakfast/
Disallow /special/
Disallow /~/media/student-academic-services/suggested-course-sequences/20134course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20145course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20156course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20167course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20178course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20189/
Disallow /~/media/student-academic-services/suggested-course-sequences/20190course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20201-course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20212-course-sequences/
Disallow /~/media/student-academic-services/suggested-course-sequences/20223-course-sequences/
Disallow /catalog/undergraduate/20234
Disallow /catalog/undergraduate/20223
Disallow /catalog/undergraduate/20212
Disallow /catalog/undergraduate/20201
Disallow /catalog/undergraduate/20190
Disallow /catalog/undergraduate/20189
Disallow /catalog/undergraduate/20178
Disallow /catalog/undergraduate/20167
Disallow /catalog/undergraduate/20156
Disallow /acad/UGCatalog/20145/
Disallow /acad/UGCatalog/20134/
Disallow /acad/UGCatalog/20123/
Disallow /acad/UGCatalog/20112/
Disallow /acad/UGCatalog/20101/
Disallow /acad/UGCatalog/20090/
Disallow /acad/UGCatalog/20089/
Disallow /acad/UGCatalog/20078/
Disallow /acad/UGCatalog/20067/
Disallow /acad/UGCatalog/20056/
Disallow /acad/UGCatalog/20045/
Disallow /acad/UGCatalog/20034/
Disallow /acad/UGCatalog/20023/
Disallow /acad/ugcatalog/20145/
Disallow /acad/ugcatalog/20134/
Disallow /acad/ugcatalog/20123/
Disallow /acad/ugcatalog/20112/
Disallow /acad/ugcatalog/20101/
Disallow /acad/ugcatalog/20090/
Disallow /acad/ugcatalog/20089/
Disallow /acad/ugcatalog/20078/
Disallow /acad/ugcatalog/20067/
Disallow /acad/ugcatalog/20056/
Disallow /acad/ugcatalog/20045/
Disallow /acad/ugcatalog/20034/
Disallow /acad/ugcatalog/20023/
Disallow /catalog/graduate/20234
Disallow /catalog/graduate/20223
Disallow /catalog/graduate/20212
Disallow /catalog/graduate/20201
Disallow /catalog/graduate/20190
Disallow /catalog/graduate/20189
Disallow /catalog/graduate/20178
Disallow /catalog/graduate/20167
Disallow /acad/GRCatalog/20178/
Disallow /acad/GRCatalog/20167/
Disallow /acad/GRcatalog/20156/
Disallow /acad/GRCatalog/20156/
Disallow /acad/GRCatalog/20145/
Disallow /acad/GRcatalog/20134/
Disallow /acad/GRCatalog/20134/
Disallow /acad/GRCatalog/20123/
Disallow /acad/GRCatalog/20112/
Disallow /acad/GRCatalog/20101/
Disallow /acad/GRcatalog/20090/
Disallow /acad/GRCatalog/20090/
Disallow /acad/GRCatalog/20089/
Disallow /acad/GRCatalog/20078/
Disallow /acad/GRCatalog/20067/
Disallow /acad/GRCatalog/20056/
Disallow /acad/GRCatalog/20045/
Disallow /acad/GRCatalog/20034/
Disallow /acad/GRCatalog/20023/
Disallow /depts/sas/catalog-archive
Disallow /university-leadership/president/presidential-news/b*
Disallow /news/eNews/assp*
Disallow /acad/grcatalog/20178/
Disallow /acad/grcatalog/20167/
Disallow /acad/grcatalog/20156/
Disallow /acad/grcatalog/20145/
Disallow /acad/grcatalog/20134/
Disallow /acad/grcatalog/20123/
Disallow /acad/grcatalog/20112/
Disallow /acad/grcatalog/20101/
Disallow /acad/grcatalog/20090/
Disallow /acad/grcatalog/20089/
Disallow /acad/grcatalog/20078/
Disallow /acad/grcatalog/20067/
Disallow /acad/grcatalog/20056/
Disallow /acad/grcatalog/20045/
Disallow /acad/grcatalog/20034/
Disallow /acad/grcatalog/20023/
Disallow /undergraduate-admissions/scholarships-financial-aid/scholarships-and-grants/distinguished-scholar-award-competition*
Disallow /undergraduate-admissions/visit/events/dinner-reception
Disallow /births.html
Disallow /footnotes/
Disallow /NewsItem.aspx?
Disallow /orgs/research/
Disallow /about-spu/events/saltandlight
Disallow /academics/school-of-business-and-economics/graduate-programs/new-mba
Disallow /graduate-admissions/application-requirements/
Back to top

Comments

  • /robots.txt file for //www.spu.edu/
  • for more information/help: //www.robotstxt.org/wc/exclusion-admin.html
  • google is always helpful: //www.google.com/bot.html#norobots
  • this statement below is saying: any web crawler (*) CANNOT access the /classifieds/ folder:
  • but they CAN still access the /webApps/ folder...
  • Removed due to old tuition rates listings - 07-14-2017 JL
  • no old suggested course sequences in sas
  • no old catalogs in search, access through nav only
  • Removed 20156 and 20167 grad catalog and scholar award disallow 01-08-2018 JL
  • Removed 20156 sitecore catalog 10-07-2016 M. G. H.
  • Removed lowercase grad catalog 09-29-2016 JL
  • No unintended submissions 01-29-2016 JL
  • No Response Footnotes 04-21-2014 JL
  • no old news pages
  • no WSRC
  • Salt and Light page only for attendees, hide from search
  • Links for Collegis review
  • if there is another folder on the server, then just add the line:
  • Disallow: /folder_name/
  • you can have subfolders as well:
  • Disallow: /folder_name/sub_folder_name/
Back to top