thecowboys.us
robots.txt
Robots Exclusion Standard data for thecowboys.us
Resource Scan
Scan Details
| Site Domain | thecowboys.us |
| Base Domain | thecowboys.us |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-06-05T04:46:51+00:00 |
| Next Scan | 2024-06-19T04:46:51+00:00 |
Last Successful Scan
| Scanned | 2024-04-28T04:43:27+00:00 |
| URL | https://thecowboys.us/robots.txt |
| Domain IPs | 104.21.92.78, 172.67.190.67, 2606:4700:3033::6815:5c4e, 2606:4700:3036::ac43:be43 |
| Response IP | 104.21.92.78 |
| Found | Yes |
| Hash | 05a6c37d2e784550fa7f4295a6dac4d6832d22deaaf0fd918c9d208f2c37574c |
| SimHash | e324dee201b3 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /viral/ |
| Allow | /wp-admin/admin-ajax.php |
| Disallow | /bspr/ |
| Disallow | /frame/ |
| Disallow | /wp-admin/ |
Other Records
| Field | Value |
|---|---|
| sitemap | http://thecowboys.us/sitemap_index.xml |
| sitemap | http://thecowboys.us/sitemap-gshu/stt-index.xml |
| sitemap | http://thecowboys.us/kwmap/sitemap.xml |
Warnings
- `host` is not a known field.