cspan.com
robots.txt
Robots Exclusion Standard data for cspan.com
Resource Scan
Scan Details
| Site Domain | cspan.com |
| Base Domain | cspan.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-08-24T18:00:31+00:00 |
| Next Scan | 2024-10-23T18:00:31+00:00 |
Last Successful Scan
| Scanned | 2024-06-26T17:59:54+00:00 |
| URL | https://cspan.com/robots.txt |
| Redirect | https://www.c-span.org:443/robots.txt |
| Redirect Domain | www.c-span.org |
| Redirect Base | c-span.org |
| Domain IPs | 108.156.133.107, 108.156.133.56, 108.156.133.72, 108.156.133.77 |
| Redirect IPs | 13.33.88.118, 13.33.88.52, 13.33.88.79, 13.33.88.94 |
| Response IP | 13.33.88.79 |
| Found | Yes |
| Hash | f940250372d24a24cf2b173d6e121912b3425df1c37b95797a3c8c8cd9c7dc91 |
| SimHash | 0c05d8c20d15 |
Groups
adbeat_bot
accompanybot
ahrefsbot
asterias
baiduspider
gigabot
gptbot
nutch
psbot
robozilla
scrubby
teoma
trendkite-akashic-crawler
twiceler
yahoo-blogs/v3.9
yahoo-mmcrawler
yandex
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /videoLibrary/assets/swf/ |
| Disallow | /assets/swf |
| Disallow | /videoLibrary/common/services/ |
| Disallow | /common/services/ |
| Disallow | /videoLibrary/transcript |
| Disallow | /transcript |
| Disallow | /videoLibrary/ajax |
| Disallow | /ajax |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 4 |