gander.wustl.edu
robots.txt
Robots Exclusion Standard data for gander.wustl.edu
Resource Scan
Scan Details
| Site Domain | gander.wustl.edu |
| Base Domain | wustl.edu |
| Scan Status | Ok |
| Last Scan | 2025-08-17T22:03:46+00:00 |
| Next Scan | 2025-09-16T22:03:46+00:00 |
Last Scan
| Scanned | 2025-08-17T22:03:46+00:00 |
| URL | https://gander.wustl.edu/robots.txt |
| Domain IPs | 128.252.93.96 |
| Response IP | 128.252.93.96 |
| Found | Yes |
| Hash | bb917f668b913f125b81a19883c40c1e9ce9f2fffbdd16f81361d1a84b7a38a1 |
| SimHash | e4334820e73f |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /admin/stats/ |
| Disallow | /goldenPath/certificate.html |
| Disallow | /goldenPath/certificates/ |
| Disallow | /cgi-bin/hgTracks*.customText*. |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |