msn.be
robots.txt
Robots Exclusion Standard data for msn.be
Resource Scan
Scan Details
Site Domain | msn.be |
Base Domain | msn.be |
Scan Status | Ok |
Last Scan | 2024-09-25T20:11:04+00:00 |
Next Scan | 2024-10-02T20:11:04+00:00 |
Last Scan
Scanned | 2024-09-25T20:11:04+00:00 |
URL | http://msn.be/robots.txt |
Redirect | http://www.msn.com/robots.txt |
Redirect Domain | www.msn.com |
Redirect Base | msn.com |
Domain IPs | 131.253.33.203 |
Redirect IPs | 204.79.197.203 |
Response IP | 204.79.197.203 |
Found | Yes |
Hash | 584c9dd32d58ecd04e5a4bb3e65baad88f47d428c899afa448b1b755fce1b8c1 |
SimHash | 4d90db71c526 |
Groups
*
Rule | Path |
---|---|
Disallow | /*/health/search/filter |
Disallow | /spartan |
Disallow | /pt-ao |
Disallow | /*preview%3D* |
Disallow | /*/autos/marketplace/product/* |
Disallow | /*/cars/marketplace/product/* |
Disallow | /*?item=* |
Disallow | /*%26item%3D* |
Disallow | /codeCoverage |
Disallow | /*/studio |
Disallow | /*?loc=*&cvid=* |
Disallow | /*/shopping/ |
yandex.com/bot
Rule | Path |
---|---|
Disallow | /*/weather/ |
Disallow | /*/clima/ |
Disallow | /*/cuaca/ |
Disallow | /*/eltiempo/ |
Disallow | /*/el-tiempo/ |
Disallow | /*/havadurumu/ |
Disallow | /*/idojaras/ |
Disallow | /*/meteo/ |
Disallow | /*/meteorologia/ |
Disallow | /*/pocasi/ |
Disallow | /*/pogoda/ |
Disallow | /*/saa/ |
Disallow | /*/vader/ |
Disallow | /*/vejr/ |
Disallow | /*/weer/ |
Disallow | /*/wetter/ |
Other Records
Field | Value |
---|---|
crawl-delay | 1 |
Other Records
Field | Value |
---|---|
sitemap | https://www.msn.com/sitemaps/casualgames/sitemap-index.xml |
sitemap | https://www.msn.com/sitemaps/consumption/index.xml |
sitemap | https://www.msn.cn/sitemaps/consumption/index.xml |
sitemap | https://www.msn.com/sitemaps/health/health-sitemap-index.xml |
sitemap | https://www.msn.com/staticsb/statics/latest/0/casualgames/sitemaps/sitemap-index.xml |
sitemap | https://www.msn.com/staticsb/statics/latest/0/finance/sitemaps/sitemap-index.xml |
sitemap | https://www.msn.com/sitemaps/weather/sitemap-index.xml |