profiles.stanford.edu
robots.txt
Robots Exclusion Standard data for profiles.stanford.edu
Resource Scan
Scan Details
| Site Domain | profiles.stanford.edu |
| Base Domain | stanford.edu |
| Scan Status | Ok |
| Last Scan | 2025-07-01T14:27:28+00:00 |
| Next Scan | 2025-07-31T14:27:28+00:00 |
Last Scan
| Scanned | 2025-07-01T14:27:28+00:00 |
| URL | https://profiles.stanford.edu/robots.txt |
| Domain IPs | 171.67.37.65 |
| Response IP | 171.67.37.65 |
| Found | Yes |
| Hash | 787e496ad7ac1285995d3826d5cdcf4f2510e7a6f8992375f3a3e5efab45f5ef |
| SimHash | 6514d0726312 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /search |
| Disallow | /proxy |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Comments