haarets.co.il
robots.txt
Robots Exclusion Standard data for haarets.co.il
Resource Scan
Scan Details
| Site Domain | haarets.co.il |
| Base Domain | haarets.co.il |
| Scan Status | Failed |
| Failure Reason | Scan timed out. |
| Last Scan | 2024-08-22T19:43:45+00:00 |
| Next Scan | 2024-11-20T19:43:45+00:00 |
Last Successful Scan
| Scanned | 2023-01-05T11:24:32+00:00 |
| URL | http://haarets.co.il/robots.txt |
| Redirect | https://www.haaretz.co.il/robots.txt |
| Redirect Domain | www.haaretz.co.il |
| Redirect Base | haaretz.co.il |
| Domain IPs | 192.118.72.27 |
| Redirect IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Response IP | 199.232.46.217 |
| Found | Yes |
| Hash | 585f0932f15d2d7793472b5236ec9b581d21a599702e7b30e01430dbaf0c5bdf |
| SimHash | cd4e53c763a6 |
Groups
*
| Rule | Path |
|---|---|
| Allow | /cmlink/tag-rss-* |
| Allow | /cmlink/author-rss-* |
| Allow | /hasite/images/Edge/* |
| Disallow | /*?cache= |
| Disallow | /*2.385* |
| Disallow | /*2.2589* |
| Disallow | /*2.2587* |
| Allow | /2.385/2.409/2.725/1.2502322 |
| Disallow | /*polopoly_fs/* |
| Disallow | /*misc/article-print-page/* |
| Disallow | /*wed/* |
| Disallow | /*tue/* |
| Disallow | /*tmc/* |
| Disallow | /*reshet-shoken/* |
| Disallow | /*thu/* |
| Disallow | /sun* |
| Disallow | /*safrut/* |
| Disallow | /*opinions/topic-of-the-day/* |
| Disallow | /*musafim/* |
| Disallow | */mon/* |
| Disallow | /*marketing/* |
| Disallow | /*hasite/* |
| Disallow | /*fri/* |
| Disallow | /*eng/* |
| Disallow | /*create_html.jhtml?itemNo=* |
| Disallow | /*common/* |
| Disallow | /*captain/spages/* |
| Disallow | */arch/* |
| Disallow | /*misc/social-reader/* |
| Disallow | /*mobile/* |
| Disallow | /*st/inter/Global/dailyedition/* |
| Disallow | /*st/inter/Hheb/paywall/scripts/bots.js |
| Disallow | /*htz/js/bots.js |
| Disallow | /*hasen/* |
| Disallow | /18083239/* |
| Disallow | /20674832/* |
| Disallow | /*misc/comment-page/* |
| Disallow | /*mobile/* |
| Disallow | /*/19.* |
| Disallow | /*/tests* |
| Disallow | /*misc/smartphone-article/* |
| Disallow | /*gallery/recipes/wine/* |
| Disallow | /*sport-ga/* |
| Disallow | /*magazine-ga/* |
| Disallow | /*literature-ga/* |
| Disallow | /*opinions-ga/* |
| Disallow | /*captain-ga/* |
| Disallow | /*opinions-ga/* |
| Disallow | /*gallery-ga/* |
| Disallow | /*news-ga/* |
| Disallow | /*blogs-ga/* |
| Disallow | /*magazine/dining/* |
| Disallow | /*magazine/recipes/* |
| Disallow | /*9401/* |
| Disallow | /*misc/comment-page |
| Disallow | /*misc/comment-page/* |
| Disallow | /*misc/search-results |
| Disallow | /*search-results |
| Disallow | /*misc/haaretzsmartphoneapp/searchresult |
| Disallow | /*misc/all-headlines |
| Disallow | /*cmlink/* |
| Disallow | /*tmr/* |
| Disallow | /*inon2* |
| Disallow | /*json/cmlink/* |
| Disallow | /*misc/haaretzsmartphoneapp* |
| Disallow | /*misc/iphone-article/* |
| Disallow | /*smadar/ |
| Disallow | /*misc/haaretzsmartphoneapp/* |
| Disallow | /*mobileApp/* |
| Disallow | /*mobile/* |
| Disallow | /*food/kolhair/* |
| Disallow | /*misc/fast-article/* |
| Disallow | /*gallery/EXT-INTERACTIVE-1.5912255 |
| Disallow | /*news/science/* |
| Disallow | /papi* |
| Disallow | /*jobs* |
| Disallow | /misc/RECIPE* |
| Disallow | /misc/RECIPE* |
| Disallow | *mobileApp/* |
| Disallow | /*_next/static/chunks* |
| Disallow | /*srv* |
| Disallow | /*ipAcl* |
| Disallow | */gql* |
| Disallow | */location-* |
| Disallow | */event-* |
| Disallow | /*event-1.5889992 |
| Disallow | /*event-1.4563026 |
| Disallow | /*event-1.4612333 |
| Disallow | /*event-1.3467058 |
| Disallow | /*location-1.3175120 |
| Disallow | /*location-1.3438636 |
| Disallow | /*1.3310970 |
| Disallow | /misc/1.1113209 |
| Disallow | /news/orly/.premium-1.2422797 |
| Disallow | *1.5906622 |
| Disallow | /*labels/1.8014627 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.haaretz.co.il/sitemap.xml |
| sitemap | https://www.haaretz.co.il/news-sitemap-content.xml |
| sitemap | https://www.haaretz.co.il/news-sitemap-latest.xml |