haaretz.com
robots.txt
Robots Exclusion Standard data for haaretz.com
Resource Scan
Scan Details
| Site Domain | haaretz.com |
| Base Domain | haaretz.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a server error. |
| Last Scan | 2024-08-25T05:36:37+00:00 |
| Next Scan | 2024-11-23T05:36:37+00:00 |
Last Successful Scan
| Scanned | 2023-02-01T14:42:12+00:00 |
| URL | https://haaretz.com/robots.txt |
| Redirect | https://www.haaretz.com/robots.txt |
| Redirect Domain | www.haaretz.com |
| Redirect Base | haaretz.com |
| Domain IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Redirect IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Response IP | 199.232.46.217 |
| Found | Yes |
| Hash | 20f644d275711326c21f475584c9ca83a6492590dd89285ba71b47a35fb0a24a |
| SimHash | db5c80d2e201 |
Groups
*
| Rule | Path |
|---|---|
| Allow | /.well-known/amphtml/apikey.pub |
*
| Rule | Path |
|---|---|
| Disallow | */mypromotions |
| Disallow | */misc/sitemap-page |
| Disallow | */misc/article-print-page/* |
| Disallow | */misc/iphone-blog-article/* |
| Disallow | */misc/comment-page/* |
| Disallow | */hasen/* |
| Disallow | */captain/* |
| Disallow | */9401/* |
| Disallow | */nyt/* |
| Disallow | */mobile/* |
| Disallow | */news/international/* |
| Disallow | /9039763/* |
| Disallow | */misc/comment-page/* |
| Disallow | */misc/adxtest1/* |
| Disallow | */misc/search-results/* |
| Disallow | */misc/search-results |
| Disallow | */misc/all-today-s-headlines/* |
| Disallow | */misc/iphone-article/* |
| Disallow | */misc/haaretzcomsmartphoneapp* |
| Disallow | */json/cmlink/* |
| Disallow | */smadar* |
| Disallow | /*2.12984* |
| Disallow | /*2.169* |
| Disallow | */papi* |
| Disallow | *1.5460918* |
| Disallow | */wires* |
| Disallow | *1.7362185* |
| Disallow | */polopoly_fs* |
| Disallow | *cmlink/* |
| Disallow | *mobileApp/* |
| Disallow | *dailybrief* |
| Disallow | /*_next/static/* |
| Disallow | /*ipAcl* |
| Disallow | *%3D1.6* |
| Disallow | */1.6* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.haaretz.com/sitemap.xml |
| sitemap | https://www.haaretz.com/news-sitemap-content.xml |
Comments