themarker.com
robots.txt
Robots Exclusion Standard data for themarker.com
Resource Scan
Scan Details
| Site Domain | themarker.com |
| Base Domain | themarker.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a server error. |
| Last Scan | 2024-02-26T09:26:13+00:00 |
| Next Scan | 2024-05-26T09:26:13+00:00 |
Last Successful Scan
| Scanned | 2023-01-30T17:53:28+00:00 |
| URL | https://themarker.com/robots.txt |
| Redirect | https://www.themarker.com/robots.txt |
| Redirect Domain | www.themarker.com |
| Redirect Base | themarker.com |
| Domain IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Redirect IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Response IP | 199.232.46.217 |
| Found | Yes |
| Hash | c0da968b586ec3a30b4cc8d797d9984f38e17ce7cf636547b232adc736a47d92 |
| SimHash | d41a5645a392 |
Groups
*
| Rule | Path |
|---|---|
| Allow | /*cmlink/tag-rss-* |
| Allow | /*cmlink/author-rss-* |
| Disallow | /*ibo/* |
| Disallow | /*2.171/* |
| Disallow | *2.385* |
| Disallow | /*ads/* |
| Disallow | /*alerts/* |
| Disallow | /*binladen/* |
| Disallow | /*eng/* |
| Disallow | /*tmc/* |
| Disallow | /*toolbar/* |
| Disallow | /*misc/article-print-page/* |
| Disallow | /*mobile/* |
| Disallow | /*q1/* |
| Disallow | /*q2/* |
| Disallow | /*q3/* |
| Disallow | /*q4/* |
| Disallow | /*hitech/* |
| Disallow | /*tent-protest/* |
| Disallow | /*9401/* |
| Disallow | /*misc/comment-page* |
| Disallow | /*cmlink/* |
| Disallow | */hrtz* |
| Disallow | /*beta/* |
| Disallow | /*misc/breaking-news* |
| Disallow | /*misc/search-results |
| Disallow | /*misc/all-headlines |
| Disallow | /*misc/smartphonexmlfeed/* |
| Disallow | /*misc/themarkersmartphoneapp* |
| Disallow | /*st/c/work/* |
| Disallow | /*smadar |
| Disallow | /*json/cmlink/* |
| Disallow | /*_next/static/chunks* |
| Disallow | /*papi* |
| Disallow | /*advisors* |
| Disallow | *mobileApp/* |
| Disallow | /*ipAcl* |
| Disallow | /law/1.524560 |
| Disallow | /law/1.473347 |
| Disallow | /law/1.380132 |
| Disallow | /labels/1.6570269 |