Log In Sign Up

techlive.wsj.com
robots.txt

Robots Exclusion Standard data for techlive.wsj.com

Resource Scan

Scan Details

Site Domain techlive.wsj.com
Base Domain wsj.com
Scan Status Ok
Last Scan2025-12-09T01:06:18+00:00
Next Scan 2026-01-08T01:06:18+00:00

Last Scan

Scanned2025-12-09T01:06:18+00:00
URL https://techlive.wsj.com/robots.txt
Domain IPs 192.0.66.219, 2a04:fa87:fffd::c000:42db
Response IP 192.0.66.219
Found Yes
Hash 56a9c55e63382d8aca92fe0c40e508142063a58a16010371812a5d63bd414ba5
SimHash f8011c09ef84

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /event/
Disallow /block/*
Disallow /resource/*
Disallow /ticket/*
Disallow /wp-admin/
Disallow /agenda/
Disallow /post-template/
Disallow /topic/
Disallow /categories/

twitterbot

Rule Path
Disallow /amp/*

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://techlive.wsj.com/sitemap_index.xml
Back to top

Comments

  • NOTICE: Collection of content and other data on https://www.wsj.com/ through
  • automated means is prohibited unless you have express written
  • permission from Dow Jones & Company, Inc. and may only be conducted for the
  • limited purpose contained in said permission.
  • Dow Jones & Company, Inc. Terms of Use may be found at
  • https://www.dowjones.com/terms-of-use/
  • If you would like to apply for permission to license the
  • intellectual property and/or other materials of Dow Jones & Company, Inc.’s
  • brands, please contact us via email at copyright@dowjones.com.
Back to top