c-spanvideo.org
robots.txt
Robots Exclusion Standard data for c-spanvideo.org
Resource Scan
Scan Details
| Site Domain | c-spanvideo.org |
| Base Domain | c-spanvideo.org |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-05-25T21:13:35+00:00 |
| Next Scan | 2024-08-23T21:13:35+00:00 |
Last Successful Scan
| Scanned | 2024-01-27T21:12:13+00:00 |
| URL | https://www.c-spanvideo.org/robots.txt |
| Domain IPs | 13.33.33.103, 13.33.33.105, 13.33.33.110, 13.33.33.120 |
| Response IP | 99.86.91.21 |
| Found | Yes |
| Hash | f940250372d24a24cf2b173d6e121912b3425df1c37b95797a3c8c8cd9c7dc91 |
| SimHash | 0c05d8c20d15 |
Groups
adbeat_bot
accompanybot
ahrefsbot
asterias
baiduspider
gigabot
gptbot
nutch
psbot
robozilla
scrubby
teoma
trendkite-akashic-crawler
twiceler
yahoo-blogs/v3.9
yahoo-mmcrawler
yandex
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /videoLibrary/assets/swf/ |
| Disallow | /assets/swf |
| Disallow | /videoLibrary/common/services/ |
| Disallow | /common/services/ |
| Disallow | /videoLibrary/transcript |
| Disallow | /transcript |
| Disallow | /videoLibrary/ajax |
| Disallow | /ajax |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 4 |