blogs.nytimes.com
robots.txt
Robots Exclusion Standard data for blogs.nytimes.com
Resource Scan
Scan Details
Site Domain | blogs.nytimes.com |
Base Domain | nytimes.com |
Scan Status | Ok |
Last Scan | 2024-05-06T00:00:32+00:00 |
Next Scan | 2024-05-20T00:00:32+00:00 |
Last Scan
Scanned | 2024-05-06T00:00:32+00:00 |
URL | https://blogs.nytimes.com/robots.txt |
Redirect | https://www.nytimes.com/robots.txt |
Redirect Domain | www.nytimes.com |
Redirect Base | nytimes.com |
Domain IPs | 151.101.1.164, 151.101.129.164, 151.101.193.164, 151.101.65.164 |
Redirect IPs | 151.101.1.164, 151.101.129.164, 151.101.193.164, 151.101.65.164 |
Response IP | 199.232.45.164 |
Found | Yes |
Hash | 9532a3ecd4beab502e9444acdd552a03dda42d8de4aa4a97922f3a8e7c03b419 |
SimHash | fc5b111e2375 |
Groups
*
Rule | Path |
---|---|
Disallow | /ads/ |
Disallow | /adx/bin/ |
Disallow | /puzzles/leaderboards/invite/* |
Disallow | /svc |
Allow | /svc/crosswords |
Allow | /svc/games |
Allow | /svc/letter-boxed |
Allow | /svc/spelling-bee |
Allow | /svc/vertex |
Allow | /svc/wordle |
Disallow | /athletic/ |
Disallow | /video/embedded/* |
Disallow | /search |
Disallow | /multiproduct/ |
Disallow | /hd/ |
Disallow | /inyt/ |
Disallow | /*?*query= |
Disallow | /*.pdf$ |
Disallow | /*?*login= |
Disallow | /*?*searchResultPosition= |
Disallow | /*?*campaignId= |
Disallow | /*?*mcubz= |
Disallow | /*?*smprod= |
Disallow | /*?*ProfileID= |
Disallow | /*?*ListingID= |
Disallow | /wirecutter/wp-admin/ |
Disallow | /wirecutter/*.zip$ |
Disallow | /wirecutter/*.csv$ |
Disallow | /wirecutter/deals/beta |
Disallow | /wirecutter/data-requests |
Disallow | /wirecutter/search |
Disallow | /wirecutter/*?s= |
Disallow | /wirecutter/*%26xid%3D |
Disallow | /wirecutter/*?q= |
Disallow | /wirecutter/*?l= |
Disallow | /search |
Disallow | /*?*smid= |
Disallow | /*?*partner= |
Disallow | /*?*utm_source= |
Allow | /wirecutter/*?*utm_source= |
Allow | /ads/public/ |
Allow | /svc/news/v3/all/pshb.rss |
Other Records
Field | Value |
---|---|
sitemap | https://www.nytimes.com/sitemaps/new/news.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/sitemap.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/collections.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/video.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/cooking.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz |
sitemap | https://www.nytimes.com/sitemaps/new/regions.xml |
sitemap | https://www.nytimes.com/sitemaps/new/best-sellers.xml |
sitemap | https://www.nytimes.com/sitemaps/www.nytimes.com/2016_election_sitemap.xml.gz |
sitemap | https://www.nytimes.com/elections/2018/sitemap |
sitemap | https://www.nytimes.com/wirecutter/sitemapindex.xml |
Comments