wsj.com
robots.txt
Robots Exclusion Standard data for wsj.com
Resource Scan
Scan Details
| Site Domain | wsj.com |
| Base Domain | wsj.com |
| Scan Status | Ok |
| Last Scan | 2024-11-01T10:27:32+00:00 |
| Next Scan | 2024-11-08T10:27:32+00:00 |
Last Scan
| Scanned | 2024-11-01T10:27:32+00:00 |
| URL | https://wsj.com/robots.txt |
| Redirect | https://www.wsj.com/robots.txt |
| Redirect Domain | www.wsj.com |
| Redirect Base | wsj.com |
| Domain IPs | 13.33.88.116, 13.33.88.43, 13.33.88.68, 13.33.88.77 |
| Redirect IPs | 13.35.210.120, 13.35.210.65, 13.35.210.75, 13.35.210.78, 2600:9000:2078:3e00:3:4b0:de80:93a1, 2600:9000:2078:4c00:3:4b0:de80:93a1, 2600:9000:2078:6600:3:4b0:de80:93a1, 2600:9000:2078:7000:3:4b0:de80:93a1, 2600:9000:2078:9200:3:4b0:de80:93a1, 2600:9000:2078:a400:3:4b0:de80:93a1, 2600:9000:2078:b600:3:4b0:de80:93a1, 2600:9000:2078:ee00:3:4b0:de80:93a1 |
| Response IP | 13.35.210.75 |
| Found | Yes |
| Hash | 88f031f0aa7f6d06f9eece81b8b0dc075c8a4e5b23db8688ec0a466b2cdd85d0 |
| SimHash | 509850df8670 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /article_email/* |
| Disallow | /user/* |
| Disallow | /pdf/documents/* |
| Disallow | /login/* |
| Disallow | /acct/* |
| Disallow | /msgcenter/* |
| Disallow | /setup/* |
| Disallow | /marketing/* |
| Disallow | /public/article/* |
| Disallow | /public/search/ |
| Disallow | /public/search* |
| Disallow | /search* |
| Disallow | /public/page/wsj-x-marketing.html |
| Disallow | /public/page/news-media-marketing.html |
| Disallow | /public/page/0_0_WP_RT_MARKETING.html |
| Disallow | /news/articles/SB2* |
| Disallow | /news/articles/SB3* |
| Disallow | /news/articles/SB4* |
| Disallow | /articles/SB2* |
| Disallow | /articles/SB3* |
| Disallow | /articles/SB4* |
| Disallow | /article/AP* |
| Disallow | /article/BT-CO* |
| Disallow | /article/DN-CO* |
| Disallow | /article/PR-CO* |
| Disallow | /article/HUG* |
| Disallow | /video/search/* |
| Disallow | /articles/BT-CO* |
| Disallow | /articles/DN-CO* |
| Disallow | /articles/PR-CO* |
| Disallow | /news/articles/BT-CO* |
| Disallow | /news/articles/DN-CO* |
| Disallow | /news/articles/PR-CO* |
| Disallow | /catchup/* |
| Disallow | /articles/the-meaning-behind-juneteenth-11592413234 |
| Disallow | /emailservice/* |
| Disallow | /emailsignup/* |
| Disallow | /insetsrv/v1/* |
| Disallow | /user/fpd/api/* |
| Disallow | /Date%28* |
| Disallow | /auth/sso/proxy-login* |
| Disallow | /client/ |
| Disallow | /buyside/search-results?*term=* |
msnptc/1.0
| Rule | Path |
|---|---|
| Disallow | /article_email/* |
| Disallow | /login/* |
| Disallow | /acct/* |
| Disallow | /msgcenter/* |
| Disallow | /setup/* |
| Disallow | /user/* |
| Disallow | /marketing/* |
| Disallow | /public/article/* |
| Disallow | /public/search/ |
| Disallow | /public/search* |
| Disallow | /search* |
| Disallow | /public/page/wsj-x-marketing.html |
| Disallow | /public/page/news-media-marketing.html |
| Disallow | /public/page/0_0_WP_RT_MARKETING.html |
| Disallow | /news/articles/SB2* |
| Disallow | /news/articles/SB3* |
| Disallow | /news/articles/SB4* |
| Disallow | /articles/SB2* |
| Disallow | /articles/SB3* |
| Disallow | /articles/SB4* |
| Disallow | /article/AP* |
| Disallow | /article/BT-CO* |
| Disallow | /article/DN-CO* |
| Disallow | /article/PR-CO* |
| Disallow | /article/HUG* |
| Disallow | /video/search/* |
| Disallow | /articles/BT-CO* |
| Disallow | /articles/DN-CO* |
| Disallow | /articles/PR-CO* |
| Disallow | /news/articles/BT-CO* |
| Disallow | /news/articles/DN-CO* |
| Disallow | /news/articles/PR-CO* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.wsj.com/sitemap.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_google_news.xml |
| sitemap | https://www.wsj.com/sitemap_topics.xml |
| sitemap | https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml |
| sitemap | https://www.wsj.com/live_news_sitemap.xml |
| sitemap | https://www.wsj.com/wsj_graphics_sitemap.xml |
| sitemap | https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_article_types.xml |
| sitemap | https://www.wsj.com/authors_sitemap.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml |
| sitemap | https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp |
| sitemap | https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp |
| sitemap | https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml |
| sitemap | https://www.wsj.com/buyside/sitemap.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/elections2024_index.xml |
Comments