xpro.mit.edu
robots.txt
Robots Exclusion Standard data for xpro.mit.edu
Resource Scan
Scan Details
| Site Domain | xpro.mit.edu |
| Base Domain | mit.edu |
| Scan Status | Ok |
| Last Scan | 2024-11-09T04:04:39+00:00 |
| Next Scan | 2024-11-23T04:04:39+00:00 |
Last Scan
| Scanned | 2024-11-09T04:04:39+00:00 |
| URL | https://xpro.mit.edu/robots.txt |
| Domain IPs | 151.101.130.133, 151.101.194.133, 151.101.2.133, 151.101.66.133, 2a04:4e42:200::645, 2a04:4e42:400::645, 2a04:4e42:600::645, 2a04:4e42::645 |
| Response IP | 151.101.2.133 |
| Found | Yes |
| Hash | 2bbcf0ee954333f71807ae3f906f94d89d98e2cd93830841bed312c5f5f8b7ab |
| SimHash | aa1782eb47f1 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /boeing |
| Disallow | /certificates/ |
| Disallow | /dashboard/ |
| Disallow | /checkout/ |
| Disallow | /ecommerce/ |
| Disallow | /signin/ |
| Disallow | /create-account/ |
| Disallow | /certificate/ |
| Disallow | /cms/ |
| Disallow | /admin/ |