newyorker.com
robots.txt
Robots Exclusion Standard data for newyorker.com
Resource Scan
Scan Details
| Site Domain | newyorker.com |
| Base Domain | newyorker.com |
| Scan Status | Ok |
| Last Scan | 2024-10-30T11:07:03+00:00 |
| Next Scan | 2024-11-06T11:07:03+00:00 |
Last Scan
| Scanned | 2024-10-30T11:07:03+00:00 |
| URL | https://newyorker.com/robots.txt |
| Redirect | https://www.newyorker.com/robots.txt |
| Redirect Domain | www.newyorker.com |
| Redirect Base | newyorker.com |
| Domain IPs | 151.101.0.239, 151.101.128.239, 151.101.192.239, 151.101.64.239 |
| Redirect IPs | 13.225.4.100, 13.225.4.32, 13.225.4.49, 13.225.4.7, 2600:9000:21b4:2a00:13:b365:2dc0:93a1, 2600:9000:21b4:4600:13:b365:2dc0:93a1, 2600:9000:21b4:9400:13:b365:2dc0:93a1, 2600:9000:21b4:9e00:13:b365:2dc0:93a1, 2600:9000:21b4:a000:13:b365:2dc0:93a1, 2600:9000:21b4:b600:13:b365:2dc0:93a1, 2600:9000:21b4:c800:13:b365:2dc0:93a1, 2600:9000:21b4:e400:13:b365:2dc0:93a1 |
| Response IP | 13.225.4.7 |
| Found | Yes |
| Hash | ede56616f5059755bdd0012de025920317f8053ad3d79da20b785a096ef805fd |
| SimHash | 101800038645 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /articles |
| Disallow | /categories |
| Disallow | /cnevideos |
| Disallow | /curatedsearches |
| Disallow | /galleries |
| Disallow | /gallery |
| Disallow | /photos |
| Disallow | /preview |
| Disallow | /projects |
| Disallow | /search |
| Disallow | /svc |
| Disallow | /wp-admin |
| Disallow | /wp-includes |
| Disallow | /*/amp |
| Disallow | /*?redirectURL= |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /cartoon/ |
| Disallow | /product/ |
| Disallow | /user-context?referrer |
| Disallow | /services.min.js?_= |
| Disallow | /com.condenast/yv8* |
googlebot
| Rule | Path |
|---|---|
| Allow | /*/amp |
| Disallow | /cartoon/ |
| Disallow | /projects |
| Disallow | /*?redirectURL= |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /product/ |
| Disallow | /search |
| Disallow | /user-context?referrer |
bingbot
| Rule | Path |
|---|---|
| Allow | /*/amp |
| Disallow | /cartoon/ |
| Disallow | /*?redirectURL= |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Burl%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /*%7B%7Bsection%7D%7D |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /product/ |
| Disallow | /search |
| Disallow | /user-context?referrer |
| Disallow | /services.min.js?_= |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.newyorker.com/sitemap.xml |
| sitemap | https://www.newyorker.com/feed/google-news-sitemap-feed/sitemap-google-news |
| sitemap | https://www.newyorker.com/contributors-sitemap.xml |
| sitemap | https://www.newyorker.com/categories-sitemap.xml |
| sitemap | https://www.newyorker.com/bundles-sitemap.xml |
| sitemap | https://www.newyorker.com/sitemap-archive-1.xml |
| sitemap | https://www.newyorker.com/sitemap-archive-2.xml |
| sitemap | https://www.newyorker.com/sitemap-archive-3.xml |
| sitemap | https://www.newyorker.com/sitemap-archive-4.xml |
| sitemap | https://www.newyorker.com/sitemap-archive-5.xml |