capecod.edu
robots.txt
Robots Exclusion Standard data for capecod.edu
Resource Scan
Scan Details
| Site Domain | capecod.edu |
| Base Domain | capecod.edu |
| Scan Status | Ok |
| Last Scan | 2024-09-08T20:13:05+00:00 |
| Next Scan | 2024-10-08T20:13:05+00:00 |
Last Scan
| Scanned | 2024-09-08T20:13:05+00:00 |
| URL | https://capecod.edu/robots.txt |
| Domain IPs | 174.129.93.217, 34.228.91.126 |
| Response IP | 174.129.93.217 |
| Found | Yes |
| Hash | 9b46457a2559ef58ba5bcdce7707e763b8cb711e35b95da62ef0f676da55c2cf |
| SimHash | 6479de164347 |
Groups
googlebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
googlebot-image
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
duckduckbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
bingbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
msnbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
funnelback
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-update/ |
terminalfour nutch spider
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /amt-gened/ |
| Disallow | /catalog-archive/ |
| Disallow | /catalog-archives/ |
| Disallow | /catalog-update/ |
*
| Rule | Path |
|---|---|
| Disallow | / |