mw1.wsj.net
robots.txt
Robots Exclusion Standard data for mw1.wsj.net
Resource Scan
Scan Details
| Site Domain | mw1.wsj.net |
| Base Domain | wsj.net |
| Scan Status | Ok |
| Last Scan | 2024-05-15T08:58:30+00:00 |
| Next Scan | 2024-06-14T08:58:30+00:00 |
Last Scan
| Scanned | 2024-05-15T08:58:30+00:00 |
| URL | https://mw1.wsj.net/robots.txt |
| Domain IPs | 108.156.133.125, 108.156.133.87, 108.156.133.97, 108.156.133.99 |
| Response IP | 108.156.133.125 |
| Found | Yes |
| Hash | f8c3c814d4976702b7d3a622b755825c404a4a10964b26e269988c3fdaf16bc1 |
| SimHash | 581d7331cf94 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /3com/ |
| Disallow | /admin/ |
| Disallow | /archive/ |
| Disallow | /bin/ |
| Disallow | /cgi-bin/ |
| Disallow | /data/ |
| Disallow | /dbcfiles/ |
| Disallow | /dhtml/ |
| Disallow | /dhtmlmenu/ |
| Disallow | /doubleclick/ |
| Disallow | /dynamiclogic/ |
| Disallow | /enf/ |
| Disallow | /feeds/ |
| Disallow | /free/ |
| Disallow | /http2_data/ |
| Disallow | /login/ |
| Disallow | /MarketPerception/ |
| Disallow | /marketperception/ |
| Disallow | /MyPortfolio/ |
| Disallow | /myportfolio/ |
| Disallow | /news/current/ |
| Disallow | /portfolio/ |
| Disallow | /RealMedia/ |
| Disallow | /realmedia/ |
| Disallow | /registration/ |
| Disallow | /tools/alerts/ |
| Disallow | /tpx/ |
| Disallow | /tools/quotes/secfilings.asp |
| Disallow | /tools/quotes/form10k.asp |
| Disallow | /tools/quotes/form10q.asp |
| Disallow | /tools/quotes/form8k.asp |
| Disallow | /tools/quotes/registrationstate.asp |
| Disallow | /tools/quotes/sched13d.asp |
| Disallow | /tools/quotes/sched13f.asp |
| Disallow | /tools/quotes/sched13g.asp |
| Disallow | /tools/quotes/secarticle.asp |
| Disallow | /wsj/ |
| Disallow | /xhtml/v/ |
msnptc/1.0
| Rule | Path |
|---|---|
| Allow | /registration/ |
| Allow | /tools/alerts/ |
| Allow | /login/ |
| Allow | /myportfolio/ |
| Disallow | /3com/ |
| Disallow | /admin/ |
| Disallow | /archive/ |
| Disallow | /bin/ |
| Disallow | /cgi-bin/ |
| Disallow | /data/ |
| Disallow | /dbcfiles/ |
| Disallow | /dhtml/ |
| Disallow | /dhtmlmenu/ |
| Disallow | /doubleclick/ |
| Disallow | /dynamiclogic/ |
| Disallow | /enf/ |
| Disallow | /feeds/ |
| Disallow | /free/ |
| Disallow | /http2_data/ |
| Disallow | /login/ |
| Disallow | /MarketPerception/ |
| Disallow | /marketperception/ |
| Disallow | /news/current/ |
| Disallow | /RealMedia/ |
| Disallow | /realmedia/ |
| Disallow | /tpx/ |
| Disallow | /tools/quotes/secfilings.asp |
| Disallow | /tools/quotes/form10k.asp |
| Disallow | /tools/quotes/form10q.asp |
| Disallow | /tools/quotes/form8k.asp |
| Disallow | /tools/quotes/registrationstate.asp |
| Disallow | /tools/quotes/sched13d.asp |
| Disallow | /tools/quotes/sched13f.asp |
| Disallow | /tools/quotes/sched13g.asp |
| Disallow | /tools/quotes/secarticle.asp |
Other Records
| Field | Value |
|---|---|
| sitemap | http://www.marketwatch.com/sitemap-search-index.xml.gz |