colby.edu
robots.txt
Robots Exclusion Standard data for colby.edu
Resource Scan
Scan Details
| Site Domain | colby.edu |
| Base Domain | colby.edu |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-07-05T06:44:18+00:00 |
| Next Scan | 2024-10-03T06:44:18+00:00 |
Last Successful Scan
| Scanned | 2023-05-16T02:38:20+00:00 |
| URL | https://colby.edu/robots.txt |
| Domain IPs | 137.146.24.72 |
| Response IP | 137.146.24.72 |
| Found | Yes |
| Hash | fe6c2800caad76d82bd6467d36934cac5dd428cf06b79843787629d03a94a293 |
| SimHash | 14f97005dd7a |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ColbyMaster/ |
| Disallow | /academics_cs/goldfarb/cvc/calendar.cfm |
| Disallow | /academics_cs/museum/search/ |
| Disallow | /academics_cs/careercenter/ |
| Disallow | /admissions_cs/ |
| Disallow | /alumni_parents_cs/legacy/ |
| Disallow | /alumni_parents_cs/parents/ |
| Disallow | /art/ |
| Disallow | /athletics_cs/ |
| Disallow | /academics_cs/acaddept/art/ |
| Disallow | /business/colbycard/ |
| Disallow | /campus_cs/clubs/coc/trips/ |
| Disallow | /commonspot/ |
| Disallow | /cgi-bin/ |
| Disallow | /classof/2001.summer/ |
| Disallow | /colby.mag/issues/2004/spring/alumni/ |
| Disallow | /college/ |
| Disallow | /college_cs/ |
| Disallow | /cpa/ |
| Disallow | /faculty/ |
| Disallow | /focus/ |
| Disallow | /echo/ |
| Disallow | /envsci/ |
| Disallow | /info.tech/ |
| Disallow | /info.tech/stats/ |
| Disallow | /inspired/ |
| Disallow | /library/ |
| Disallow | /museum/ |
| Disallow | /music/ |
| Disallow | /news/feeds/ |
| Disallow | /news_events/c/ |
| Disallow | /news_events/calendar/ |
| Disallow | /news_events/directory/ |
| Disallow | /news_events/tags/ |
| Disallow | /off-campus/ |
| Disallow | /presidential-search/ |
| Disallow | /psychology/ |
| Disallow | /reload/ |
| Disallow | /security/ |
| Disallow | /personnel/ |
| Disallow | /slideshow/ |