nystateofpolitics.com
robots.txt
Robots Exclusion Standard data for nystateofpolitics.com
Resource Scan
Scan Details
| Site Domain | nystateofpolitics.com |
| Base Domain | nystateofpolitics.com |
| Scan Status | Ok |
| Last Scan | 2024-11-14T21:53:11+00:00 |
| Next Scan | 2024-11-21T21:53:11+00:00 |
Last Scan
| Scanned | 2024-11-14T21:53:11+00:00 |
| URL | https://nystateofpolitics.com/robots.txt |
| Domain IPs | 107.23.170.108, 34.228.148.215, 44.207.237.27, 52.203.24.99 |
| Response IP | 44.207.237.27 |
| Found | Yes |
| Hash | c8fd85b2aca4d4131f97599902e4ee535c8b617a176b7c66445bf7c12b864482 |
| SimHash | 6b2ca973e39f |
Groups
*
| Rule | Path |
|---|---|
| Allow | /$ |
| Allow | /state-of-politics/new-york |
| Allow | /state-of-politics/new-york/* |
| Allow | /sitemap.xml |
| Allow | /content/* |
| Allow | /services/* |
| Allow | /etc/* |
| Disallow | /* |
| Disallow | /*/*/partner-content/* |
| Disallow | /content/news/stories/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://nystateofpolitics.com/sitemap.xml |
Comments