online.wsj.com
robots.txt
Robots Exclusion Standard data for online.wsj.com
Resource Scan
Scan Details
| Site Domain | online.wsj.com |
| Base Domain | wsj.com |
| Scan Status | Ok |
| Last Scan | 2024-05-07T23:08:55+00:00 |
| Next Scan | 2024-05-14T23:08:55+00:00 |
Last Scan
| Scanned | 2024-05-07T23:08:55+00:00 |
| URL | https://online.wsj.com/robots.txt |
| Redirect | https://www.wsj.com/robots.txt |
| Redirect Domain | www.wsj.com |
| Redirect Base | wsj.com |
| Domain IPs | 13.33.88.116, 13.33.88.43, 13.33.88.68, 13.33.88.77, 2600:9000:223b:5200:19:29eb:8080:93a1, 2600:9000:223b:5600:19:29eb:8080:93a1, 2600:9000:223b:9800:19:29eb:8080:93a1, 2600:9000:223b:a200:19:29eb:8080:93a1, 2600:9000:223b:b200:19:29eb:8080:93a1, 2600:9000:223b:b800:19:29eb:8080:93a1, 2600:9000:223b:c400:19:29eb:8080:93a1, 2600:9000:223b:e200:19:29eb:8080:93a1 |
| Redirect IPs | 18.155.68.10, 18.155.68.110, 18.155.68.70, 18.155.68.76, 2600:9000:23d2:1000:3:4b0:de80:93a1, 2600:9000:23d2:1400:3:4b0:de80:93a1, 2600:9000:23d2:2600:3:4b0:de80:93a1, 2600:9000:23d2:5800:3:4b0:de80:93a1, 2600:9000:23d2:8400:3:4b0:de80:93a1, 2600:9000:23d2:8800:3:4b0:de80:93a1, 2600:9000:23d2:ea00:3:4b0:de80:93a1, 2600:9000:23d2:f600:3:4b0:de80:93a1 |
| Response IP | 18.155.68.76 |
| Found | Yes |
| Hash | 6952e1da170c2891cfce83e40ec515d3abe0b9ff77593f527c89ef4a516e0ed9 |
| SimHash | 758778fbd4a0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /article_email/* |
| Disallow | /user/* |
| Disallow | /pdf/documents/* |
| Disallow | /login/* |
| Disallow | /acct/* |
| Disallow | /msgcenter/* |
| Disallow | /setup/* |
| Disallow | /marketing/* |
| Disallow | /public/article/* |
| Disallow | /public/search/ |
| Disallow | /public/search* |
| Disallow | /search* |
| Disallow | /public/page/wsj-x-marketing.html |
| Disallow | /public/page/news-media-marketing.html |
| Disallow | /public/page/0_0_WP_RT_MARKETING.html |
| Disallow | /news/articles/SB2* |
| Disallow | /news/articles/SB3* |
| Disallow | /news/articles/SB4* |
| Disallow | /articles/SB2* |
| Disallow | /articles/SB3* |
| Disallow | /articles/SB4* |
| Disallow | /article/AP* |
| Disallow | /article/BT-CO* |
| Disallow | /article/DN-CO* |
| Disallow | /article/PR-CO* |
| Disallow | /article/HUG* |
| Disallow | /video/search/* |
| Disallow | /articles/BT-CO* |
| Disallow | /articles/DN-CO* |
| Disallow | /articles/PR-CO* |
| Disallow | /news/articles/BT-CO* |
| Disallow | /news/articles/DN-CO* |
| Disallow | /news/articles/PR-CO* |
| Disallow | /catchup/* |
| Disallow | /articles/the-meaning-behind-juneteenth-11592413234 |
| Disallow | /emailservice/* |
| Disallow | /emailsignup/* |
| Disallow | /insetsrv/v1/* |
| Disallow | /user/fpd/api/* |
| Disallow | /Date%28* |
| Disallow | /auth/sso/proxy-login* |
| Disallow | /client/ |
| Disallow | /*?*term=%E5%BE%B7* |
| Disallow | /*?*term=%E5%8D%B0* |
| Disallow | /*?*term=%E5%8A%9E* |
| Disallow | /*?*term=%E8%A5%BF* |
| Disallow | /*?*term=%E5%88%9A* |
| Disallow | /*?*term=%E7%BD%91* |
| Disallow | /*?*term=%E7%84%A6* |
| Disallow | /*?*term=%E6%BE%B3* |
| Disallow | /*?*term=%E5%88%B6* |
| Disallow | /*?*term=%E4%BB%BF* |
| Disallow | /*?*term=%E6%8A%A4* |
| Disallow | /*?*term=%E9%9B%85* |
| Disallow | /*?*term=%E5%85%A5* |
| Disallow | /*?*term=%E8%AE%A2* |
| Disallow | /*?*term=%E5%81%87* |
| Disallow | /*?*term=%E7%9A%84* |
| Disallow | /*?*term=%E4%B8%80* |
| Disallow | /*?*term=%E6%98%AF* |
| Disallow | /*?*term=%E4%B8%8D* |
| Disallow | /*?*term=%E4%BA%86* |
| Disallow | /*?*term=%E4%BA%BA* |
| Disallow | /*?*term=%E6%88%91* |
| Disallow | /*?*term=%E5%9C%A8* |
| Disallow | /*?*term=%E6%9C%89* |
| Disallow | /*?*term=%E4%BB%96* |
| Disallow | /*?*term=%E8%BF%99* |
| Disallow | /*?*term=%E4%B8%BA* |
| Disallow | /*?*term=%E4%B9%8B* |
| Disallow | /*?*term=%E5%A4%A7* |
| Disallow | /*?*term=%E6%9D%A5* |
| Disallow | /*?*term=%E4%BB%A5* |
| Disallow | /*?*term=%E4%B8%AA* |
| Disallow | /*?*term=%E4%B8%AD* |
| Disallow | /*?*term=%E4%B8%8A* |
| Disallow | /*?*term=%E4%BB%AC* |
| Disallow | /*?*term=%E5%88%B0* |
| Disallow | /*?*term=%E8%AF%B4* |
| Disallow | /*?*term=%E5%9B%BD* |
| Disallow | /*?*term=%E5%92%8C* |
| Disallow | /*?*term=%E5%9C%B0* |
| Disallow | /*?*term=%E4%B9%9F* |
| Disallow | /*?*term=%E5%AD%90* |
| Disallow | /*?*term=%E6%97%B6* |
| Disallow | /*?*term=%E9%81%93* |
| Disallow | /*?*term=%E5%87%BA* |
| Disallow | /*?*term=%E8%80%8C* |
| Disallow | /*?*term=%E8%A6%81* |
| Disallow | /*?*term=%E4%BA%8E* |
| Disallow | /*?*term=%E5%B0%B1* |
| Disallow | /*?*term=%E4%B8%8B* |
| Disallow | /*?*term=%E5%BE%97* |
| Disallow | /*?*term=%E5%8F%AF* |
| Disallow | /*?*term=%E4%BD%A0* |
| Disallow | /*?*term=%E5%B9%B4* |
| Disallow | /*?*term=%E7%94%9F* |
| Disallow | /*?*term=%E8%87%AA* |
| Disallow | /*?*term=%E4%BC%9A* |
| Disallow | /*?*term=%E9%82%A3* |
| Disallow | /*?*term=%E5%90%8E* |
| Disallow | /*?*term=%E8%83%BD* |
| Disallow | /*?*term=%E5%AF%B9* |
| Disallow | /*?*term=%E7%9D%80* |
| Disallow | /*?*term=%E4%BA%8B* |
| Disallow | /*?*term=%E5%85%B6* |
| Disallow | /*?*term=%E9%87%8C* |
| Disallow | /*?*term=%E6%89%80* |
| Disallow | /*?*term=%E5%8E%BB* |
| Disallow | /*?*term=%E8%A1%8C* |
| Disallow | /*?*term=%E8%BF%87* |
| Disallow | /*?*term=%E5%AE%B6* |
| Disallow | /*?*term=%E5%8D%81* |
| Disallow | /*?*term=%E7%94%A8* |
| Disallow | /*?*term=%E5%8F%91* |
| Disallow | /*?*term=%E5%A4%A9* |
| Disallow | /*?*term=%E5%A6%82* |
| Disallow | /*?*term=%E7%84%B6* |
| Disallow | /*?*term=%E4%BD%9C* |
| Disallow | /*?*term=%E6%96%B9* |
| Disallow | /*?*term=%E6%88%90* |
| Disallow | /*?*term=%E8%80%85* |
| Disallow | /*?*term=%E5%A4%9A* |
| Disallow | /*?*term=%E6%97%A5* |
| Disallow | /*?*term=%E9%83%BD* |
| Disallow | /*?*term=%E4%B8%89* |
| Disallow | /*?*term=%E5%B0%8F* |
| Disallow | /*?*term=%E5%86%9B* |
| Disallow | /*?*term=%E4%BA%8C* |
| Disallow | /*?*term=%E6%97%A0* |
| Disallow | /*?*term=%E5%90%8C* |
| Disallow | /*?*term=%E4%B9%88* |
| Disallow | /*?*term=%E7%BB%8F* |
| Disallow | /*?*term=%E6%B3%95* |
| Disallow | /*?*term=%E5%BD%93* |
| Disallow | /*?*term=%E8%B5%B7* |
| Disallow | /*?*term=%E4%B8%8E* |
| Disallow | /*?*term=%E5%A5%BD* |
| Disallow | /*?*term=%E7%9C%8B* |
| Disallow | /*?*term=%E5%AD%A6* |
| Disallow | /*?*term=%E8%BF%9B* |
| Disallow | /*?*term=%E7%A7%8D* |
| Disallow | /*?*term=%E5%B0%86* |
| Disallow | /*?*term=%E8%BF%98* |
| Disallow | /*?*term=%E5%88%86* |
| Disallow | /*?*term=%E6%AD%A4* |
| Disallow | /*?*term=%E5%BF%83* |
| Disallow | /*?*term=%E5%89%8D* |
| Disallow | /*?*term=%E9%9D%A2* |
| Disallow | /*?*term=%E5%8F%88* |
| Disallow | /*?*term=%E5%AE%9A* |
| Disallow | /*?*term=%E8%A7%81* |
| Disallow | /*?*term=%E5%8F%AA* |
| Disallow | /*?*term=%E4%B8%BB* |
| Disallow | /*?*term=%E6%B2%A1* |
| Disallow | /*?*term=%E5%85%AC* |
| Disallow | /*?*term=%E4%BB%8E* |
msnptc/1.0
| Rule | Path |
|---|---|
| Disallow | /article_email/* |
| Disallow | /login/* |
| Disallow | /acct/* |
| Disallow | /msgcenter/* |
| Disallow | /setup/* |
| Disallow | /user/* |
| Disallow | /marketing/* |
| Disallow | /public/article/* |
| Disallow | /public/search/ |
| Disallow | /public/search* |
| Disallow | /search* |
| Disallow | /public/page/wsj-x-marketing.html |
| Disallow | /public/page/news-media-marketing.html |
| Disallow | /public/page/0_0_WP_RT_MARKETING.html |
| Disallow | /news/articles/SB2* |
| Disallow | /news/articles/SB3* |
| Disallow | /news/articles/SB4* |
| Disallow | /articles/SB2* |
| Disallow | /articles/SB3* |
| Disallow | /articles/SB4* |
| Disallow | /article/AP* |
| Disallow | /article/BT-CO* |
| Disallow | /article/DN-CO* |
| Disallow | /article/PR-CO* |
| Disallow | /article/HUG* |
| Disallow | /video/search/* |
| Disallow | /articles/BT-CO* |
| Disallow | /articles/DN-CO* |
| Disallow | /articles/PR-CO* |
| Disallow | /news/articles/BT-CO* |
| Disallow | /news/articles/DN-CO* |
| Disallow | /news/articles/PR-CO* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.wsj.com/sitemap.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_google_news.xml |
| sitemap | https://www.wsj.com/sitemap_topics.xml |
| sitemap | https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml |
| sitemap | https://www.wsj.com/live_news_sitemap.xml |
| sitemap | https://www.wsj.com/wsj_graphics_sitemap.xml |
| sitemap | https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_article_types.xml |
| sitemap | https://www.wsj.com/authors_sitemap.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml |
| sitemap | https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp |
| sitemap | https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp |
| sitemap | https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_cb_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_ai_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_debt_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_cyber_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_vc_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_pe_google_news.xml |
| sitemap | https://www.wsj.com/wsjsitemaps/wsj_pro_sb_google_news.xml |
| sitemap | https://www.wsj.com/buyside/sitemaps/wsjbuyside_sitemap_index.xml |
Comments