Log In Sign Up

med.stanford.edu
robots.txt

Robots Exclusion Standard data for med.stanford.edu

Resource Scan

Scan Details

Site Domain med.stanford.edu
Base Domain stanford.edu
Scan Status Ok
Last Scan2025-06-24T21:36:13+00:00
Next Scan 2025-07-24T21:36:13+00:00

Last Scan

Scanned2025-06-24T21:36:13+00:00
URL https://med.stanford.edu/robots.txt
Domain IPs 34.117.178.225
Response IP 34.117.178.225
Found Yes
Hash 6ed42d13f78504b54f3e62ac786704ef0ba18ca8692aab1071b5bcac220caab2
SimHash 871b8530cb99

Groups

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

Rule Path
Disallow /_baks/
Disallow /_mm/
Disallow /_notes/
Disallow /about/events/stanford-medicine-live/StanfordMed_LIVE_8-23-22.html
Disallow /about_photo/archive/
Disallow /apamsa/
Disallow /appdeploy/
Disallow /appleheartstudy/1.2faq.html
Disallow /aweekinthelife/prasanna_ananth/
Disallow /Architext/
Disallow /businesscontinuity/
Disallow /careercenter3/
Disallow /center/communications/
Disallow /center/development/
Disallow /cibsr/cibsronly/
Disallow /conflict/
Disallow /covid19/app/
Disallow /demo/
Disallow /develop/
Disallow /dfa/UHC_physicians_RM_guide.pdf
Disallow /e-portfolio/
Disallow /facultysenate/
Disallow /five_questions/archive/
Disallow /healthlibrary/
Disallow /homedemo/
Disallow /homestaging/
Disallow /homestaging2/
Disallow /identity/how-we-look/*
Disallow /Images/
Disallow /irt/admin/
Disallow /irt/pagers/
Disallow /irt/web/site_tools/videoserver.html
Disallow /irt/wireless/
Disallow /lkc/
Disallow /maintenance/
Disallow /MedCenter/
Disallow /meddev/
Disallow /medredesign/
Disallow /medstaging/
Disallow /MedSchool/
Disallow /medstaging/
Disallow /MMWIP/
Disallow /new_subdirectory/
Disallow /OLT/
Disallow /osa/
Disallow /pager/
Disallow /paging/
Disallow /planning/SMP_Shuttle_Schedule.xls
Disallow /profiles/postdocs/researcher/Ruchi_Bajpai/
Disallow /profiles/frdActionServlet?choiceId=facProfile&fid=10000
Disallow /profiles/viewCV?facultyId=10168&name=William%2BWei-Lin_Tseng
Disallow /profiles/ortho/researcher/Scott_Soltys/
Disallow /profiles-test/
Disallow /protomed/
Disallow /school/gastrohep/
Disallow /school/immunology/
Disallow /school/psychiatry/PSTreatLab/
Disallow /school/Psychiatry/scn/
Disallow /school/structuralbio/
Disallow /senate/98-99/
Disallow /senate/99-00/
Disallow /senate/00-01/
Disallow /senate/01-02/
Disallow /senate/02-03/
Disallow /senate/03-04/
Disallow /senate/CRCbull/
Disallow /shc/
Disallow /sm/
Disallow /smdev/
Disallow /smi/projects/protege/download/old-releases
Disallow /smstaging/
Disallow /SPIRIT/files/
Disallow /spotlight/
Disallow /Templates/
Disallow /test/
Disallow /tds/
Disallow /vantage_point/archive/
Disallow /webtraining/
Disallow /yoursite/
Disallow /smile/
Disallow /profiles/ortho/researcher/Scott_Soltys/
Disallow /profiles/ortho/frdActionServlet?choiceId=facProfile&fid=6967
Disallow /ism/2011/january/nose-0124.html
Disallow /profiles/Vyjeyanthi_Periyakoil/
Disallow /profiles/Vyjeyanthi_Periyakoil/%3Bjsessionid%3D4574780DB8A534599650BF23A39FE9E4.tc-cap-08
Disallow /profiles/frdActionServlet?choiceId=similarPeopleSearch
Disallow /profiles/frdActionServlet?choiceId=printerprofile
Disallow /profiles/hemoncstem/faculty/Kathleen_Sakamoto/
Disallow */Templates/
Disallow */new_subdirectory/
Disallow */includes/
Disallow *1col_stationery.html
Disallow *2col_stationery.html
Disallow *3col_stationery.html
Disallow *slides.xml
Disallow */Temporary/
Disallow */temporary/
Disallow */tmp/
Disallow /content/dam/sm/radcancerbio/resources-secure/
Disallow /content/dam/sm/ccto/sunet_id_resources/

Other Records

Field Value
crawl-delay 7
Back to top

Comments

  • 80Legs search crawler. 5-25-2012: We've been informed by 80legs through a support request that they do not honor crawl-delay, but we can have them lower their request rate by opening a case with them ( http://www.80legs.com/contact-linked.html ). We are leaving the crawl-delay in the robots.txt in hopes that they will honor it in the future.
  • Baidu Search Engine Robot
  • Microsoft Search Engine Robot
  • Bing Search Engine Robot
  • Other Search Engine Robots
Back to top

Warnings

  • 3 invalid lines.
Back to top