interactive.wsj.com
robots.txt

Robots Exclusion Standard data for interactive.wsj.com

Resource Scan

Scan Details

Site Domain interactive.wsj.com
Base Domain wsj.com
Scan Status Ok
Last Scan2024-11-14T16:54:09+00:00
Next Scan 2024-11-21T16:54:09+00:00

Last Scan

Scanned2024-11-14T16:54:09+00:00
URL https://interactive.wsj.com/robots.txt
Redirect https://www.wsj.com/robots.txt
Redirect Domain www.wsj.com
Redirect Base wsj.com
Domain IPs 3.165.82.122, 3.165.82.129, 3.165.82.47, 3.165.82.57
Redirect IPs 13.35.210.120, 13.35.210.65, 13.35.210.75, 13.35.210.78, 2600:9000:2078:1c00:3:4b0:de80:93a1, 2600:9000:2078:3600:3:4b0:de80:93a1, 2600:9000:2078:6200:3:4b0:de80:93a1, 2600:9000:2078:6c00:3:4b0:de80:93a1, 2600:9000:2078:8600:3:4b0:de80:93a1, 2600:9000:2078:a400:3:4b0:de80:93a1, 2600:9000:2078:ca00:3:4b0:de80:93a1, 2600:9000:2078:da00:3:4b0:de80:93a1
Response IP 13.35.210.78
Found Yes
Hash 88f031f0aa7f6d06f9eece81b8b0dc075c8a4e5b23db8688ec0a466b2cdd85d0
SimHash 509850df8670

Groups

*

Rule Path
Disallow /article_email/*
Disallow /user/*
Disallow /pdf/documents/*
Disallow /login/*
Disallow /acct/*
Disallow /msgcenter/*
Disallow /setup/*
Disallow /marketing/*
Disallow /public/article/*
Disallow /public/search/
Disallow /public/search*
Disallow /search*
Disallow /public/page/wsj-x-marketing.html
Disallow /public/page/news-media-marketing.html
Disallow /public/page/0_0_WP_RT_MARKETING.html
Disallow /news/articles/SB2*
Disallow /news/articles/SB3*
Disallow /news/articles/SB4*
Disallow /articles/SB2*
Disallow /articles/SB3*
Disallow /articles/SB4*
Disallow /article/AP*
Disallow /article/BT-CO*
Disallow /article/DN-CO*
Disallow /article/PR-CO*
Disallow /article/HUG*
Disallow /video/search/*
Disallow /articles/BT-CO*
Disallow /articles/DN-CO*
Disallow /articles/PR-CO*
Disallow /news/articles/BT-CO*
Disallow /news/articles/DN-CO*
Disallow /news/articles/PR-CO*
Disallow /catchup/*
Disallow /articles/the-meaning-behind-juneteenth-11592413234
Disallow /emailservice/*
Disallow /emailsignup/*
Disallow /insetsrv/v1/*
Disallow /user/fpd/api/*
Disallow /Date%28*
Disallow /auth/sso/proxy-login*
Disallow /client/
Disallow /buyside/search-results?*term=*

msnptc/1.0

Rule Path
Disallow /article_email/*
Disallow /login/*
Disallow /acct/*
Disallow /msgcenter/*
Disallow /setup/*
Disallow /user/*
Disallow /marketing/*
Disallow /public/article/*
Disallow /public/search/
Disallow /public/search*
Disallow /search*
Disallow /public/page/wsj-x-marketing.html
Disallow /public/page/news-media-marketing.html
Disallow /public/page/0_0_WP_RT_MARKETING.html
Disallow /news/articles/SB2*
Disallow /news/articles/SB3*
Disallow /news/articles/SB4*
Disallow /articles/SB2*
Disallow /articles/SB3*
Disallow /articles/SB4*
Disallow /article/AP*
Disallow /article/BT-CO*
Disallow /article/DN-CO*
Disallow /article/PR-CO*
Disallow /article/HUG*
Disallow /video/search/*
Disallow /articles/BT-CO*
Disallow /articles/DN-CO*
Disallow /articles/PR-CO*
Disallow /news/articles/BT-CO*
Disallow /news/articles/DN-CO*
Disallow /news/articles/PR-CO*

twitterbot

Rule Path
Disallow /amp/*

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.wsj.com/sitemap.xml
sitemap https://www.wsj.com/wsjsitemaps/wsj_google_news.xml
sitemap https://www.wsj.com/sitemap_topics.xml
sitemap https://www.wsj.com/sitemaps/web/wsj/en/sitemap_wsj_en_index.xml
sitemap https://www.wsj.com/live_news_sitemap.xml
sitemap https://www.wsj.com/wsj_graphics_sitemap.xml
sitemap https://www.wsj.com/sitemaps/web/wsj/en/sitemap_news_archive_index.xml
sitemap https://www.wsj.com/wsjsitemaps/wsj_article_types.xml
sitemap https://www.wsj.com/authors_sitemap.xml
sitemap https://www.wsj.com/wsjsitemaps/wsj_article_list_sitemap.xml
sitemap https://www.wsj.com/api-video/sitemaps/google/sitemap-google-news-video-wsj-en.asp
sitemap https://www.wsj.com/api-video/sitemaps/google/sitemap-google-video-wsj-en.asp
sitemap https://www.wsj.com/sitemaps/web/video/en/sitemap_video_en_index.xml
sitemap https://www.wsj.com/buyside/sitemap.xml
sitemap https://www.wsj.com/wsjsitemaps/elections2024_index.xml

Comments

  • For Buyside Search Results