cardinalalumni.stanford.edu
robots.txt

Robots Exclusion Standard data for cardinalalumni.stanford.edu

Resource Scan

Scan Details

Site Domain cardinalalumni.stanford.edu
Base Domain stanford.edu
Scan Status Ok
Last Scan2024-04-26T16:39:03+00:00
Next Scan 2024-05-26T16:39:03+00:00

Last Scan

Scanned2024-04-26T16:39:03+00:00
URL https://cardinalalumni.stanford.edu/robots.txt
Domain IPs 171.67.46.135
Response IP 171.67.46.135
Found Yes
Hash be50fd1577f04f0f6075abf28f8fc1007cd3c97ed07a6ecc90013b5b933aff4d
SimHash f014a41e4bf1

Groups

gigabot

Rule Path
Disallow /

ahrefsbot
huaweisymantecspider
shopwiki
yandex
moget
ichiro
naverbot
yeti
sogou spider
youdaobot
accelobot
atraxsolutions
naverbot
presans
presansbot
yacybot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

*

Rule Path
Disallow /cas/
Disallow /eventreg/
Disallow /eventsetup/
Disallow /eventsetup2/
Disallow /farmhouse/
Disallow /feedback/
Disallow /feedback2/
Disallow /get/scripts/
Disallow /giving/home
Disallow /get/layout/g2s/
Disallow /get/page/events/who-is-coming*
Disallow /get/page/g2s/
Disallow /get/page/groups/ClubLeaderResource
Disallow /get/page/regions/events
Disallow /get/page/regions/events*
Disallow /get/page/regions/events/
Disallow /get/page/regions/events/*
Disallow /get/page/regions/landing/
Disallow /get/page/test/
Disallow /giving/menu.js
Disallow /giving/part/tsc/progress_challenge.html
Disallow /giving/part/tsc/progress_excellence.html
Disallow /giving/part/tsc/progress_leaders.html
Disallow /giving/part/tsc/progress_solutions.html
Disallow /giving/part/tsf/ReunionCurrent.html
Disallow /membership/
Disallow /odr/
Disallow /pbooks/help/
Disallow /pgw/help/
Disallow /pgw/footer/
Disallow /pgw/js/
Disallow /pgw/odrdemo/
Disallow /pgw/scripts/
Disallow /pgw/test/
Disallow /pgw/saa/
Disallow /profile/
Disallow /WebDirectory/
Disallow /whoscoming/
Disallow /SmartMail/
Disallow /get/page/events?fromDate=*

Comments

  • Gigabot is listed separately, as it is reputed to mis-parse the file otherwise
  • Russian Search engine
  • Japan-based search engines
  • S Korean
  • Second-tier Chinese Search engines (Primary is Baidu, which we will allow)
  • per Jeremy on Sept 13 2010 disallow these crawlers on our site
  • allow no crawlers to browse the areas listed below