pitchfork.com
robots.txt
Robots Exclusion Standard data for pitchfork.com
Resource Scan
Scan Details
| Site Domain | pitchfork.com |
| Base Domain | pitchfork.com |
| Scan Status | Ok |
| Last Scan | 2024-06-22T16:27:05+00:00 |
| Next Scan | 2024-06-29T16:27:05+00:00 |
Last Scan
| Scanned | 2024-06-22T16:27:05+00:00 |
| URL | https://pitchfork.com/robots.txt |
| Domain IPs | 13.33.30.13, 13.33.30.19, 13.33.30.43, 13.33.30.8, 2600:9000:2200:0:1a:1603:8940:93a1, 2600:9000:2200:600:1a:1603:8940:93a1, 2600:9000:2200:6600:1a:1603:8940:93a1, 2600:9000:2200:7e00:1a:1603:8940:93a1, 2600:9000:2200:9e00:1a:1603:8940:93a1, 2600:9000:2200:b600:1a:1603:8940:93a1, 2600:9000:2200:ea00:1a:1603:8940:93a1, 2600:9000:2200:f000:1a:1603:8940:93a1 |
| Response IP | 13.33.30.43 |
| Found | Yes |
| Hash | c8da4c0cbb02418a8899427da17fbbbe0089e1daa9d34b117cd7c026963eaf13 |
| SimHash | 451091438713 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /search/ |
| Disallow | /search |
| Disallow | /*?redirectURL= |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /product/ |
| Disallow | /user-context?referrer |
| Disallow | /services.min.js?_= |
| Disallow | /preview/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://pitchfork.com/sitemap.xml |
| sitemap | https://pitchfork.com/branded-sitemap.xml |
| sitemap | https://pitchfork.com/feed/google-latest-news/sitemap-google-news |
| sitemap | https://pitchfork.com/feed/sitemap |
| sitemap | https://pitchfork.com/sitemap-archive-1.xml |
| sitemap | https://pitchfork.com/bundles-sitemap.xml |