habr.ru
robots.txt
Robots Exclusion Standard data for habr.ru
Resource Scan
Scan Details
| Site Domain | habr.ru |
| Base Domain | habr.ru |
| Scan Status | Ok |
| Last Scan | 2024-11-11T11:10:40+00:00 |
| Next Scan | 2024-11-18T11:10:40+00:00 |
Last Scan
| Scanned | 2024-11-11T11:10:40+00:00 |
| URL | https://habr.ru/robots.txt |
| Redirect | https://habr.com/robots.txt |
| Redirect Domain | habr.com |
| Redirect Base | habr.com |
| Domain IPs | 178.248.233.33 |
| Redirect IPs | 178.248.237.68 |
| Response IP | 178.248.237.68 |
| Found | Yes |
| Hash | fcadd23a8d60ba4237991bd5cdc3163cc2ebd86c073441d69923ab621c846b21 |
| SimHash | 6e42dec2b091 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /search/ |
| Disallow | /ru/search/ |
| Disallow | /en/search/ |
| Disallow | /*?*utm_source= |
| Disallow | /*?*utm_medium= |
| Disallow | /*?*utm_term= |
| Disallow | /*?*utm_campaign= |
slurp
| Rule | Path |
|---|---|
| Disallow | /search/ |
| Disallow | /ru/search/ |
| Disallow | /en/search/ |
| Disallow | /*?*utm_ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 8 |
*
| Rule | Path |
|---|---|
| Disallow | /search/ |
| Disallow | /ru/search/ |
| Disallow | /en/search/ |
| Disallow | /*?*utm_ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Warnings
- `clean-param` is not a known field.