cbs58.com
robots.txt
Robots Exclusion Standard data for cbs58.com
Resource Scan
Scan Details
| Site Domain | cbs58.com |
| Base Domain | cbs58.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-04-25T05:34:56+00:00 |
| Next Scan | 2024-05-25T05:34:56+00:00 |
Last Successful Scan
| Scanned | 2024-03-27T05:23:14+00:00 |
| URL | https://cbs58.com/robots.txt |
| Domain IPs | 104.26.6.219, 104.26.7.219, 172.67.72.146, 2606:4700:20::681a:6db, 2606:4700:20::681a:7db, 2606:4700:20::ac43:4892 |
| Response IP | 104.26.7.219 |
| Found | Yes |
| Hash | 6319d4e613c4bb8632058ea6f3d604da550982ba3a2cd7f8a3ae4c48df88f166 |
| SimHash | 4910fd55c55b |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /category/ |
| Disallow | /clip/ |
| Disallow | /link/ |
| Disallow | /news/local-news/ |
| Disallow | /news/local/ |
| Disallow | /news/national-world/ |
| Disallow | /news/crime/ |
| Disallow | /news/top-stories/ |
| Disallow | /ads/ |
| Disallow | /blogs/ |
| Disallow | /features/ |
| Disallow | /internal?* |
| Disallow | /blogs?* |
| Disallow | /video?clip* |
| Disallow | /index.php* |
| Disallow | /*.html$ |
| Disallow | /search/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.cbs58.com/CBS58.sitemap.xml |