the-village.ru
robots.txt
Robots Exclusion Standard data for the-village.ru
Resource Scan
Scan Details
Site Domain | the-village.ru |
Base Domain | the-village.ru |
Scan Status | Ok |
Last Scan | 2024-06-01T14:27:54+00:00 |
Next Scan | 2024-06-08T14:27:54+00:00 |
Last Scan
Scanned | 2024-06-01T14:27:54+00:00 |
URL | https://the-village.ru/robots.txt |
Redirect | https://www.the-village.ru/robots.txt |
Redirect Domain | www.the-village.ru |
Redirect Base | the-village.ru |
Domain IPs | 135.181.66.25 |
Redirect IPs | 135.181.66.25 |
Response IP | 135.181.66.25 |
Found | Yes |
Hash | eed5d40a84dba78ce1954f59b5328a4cb9ac145df3aab6d7f1fa4f551694eacf |
SimHash | 2b24b471eab1 |
Groups
twitterbot
Rule | Path |
---|---|
Allow | /*?utm_* |
Disallow | /search |
Disallow | /users/*/settings |
Disallow | */new$ |
Disallow | */edit$ |
Disallow | *.json$ |
Disallow | /admin/ |
Disallow | /grid/ |
Disallow | /api/users/sign_in |
Disallow | /api/ |
*
Rule | Path |
---|---|
Disallow | /comments |
Disallow | /search |
Disallow | /users/*/settings |
Disallow | */new$ |
Disallow | */edit$ |
Disallow | *.json$ |
Disallow | /*?utm_* |
Disallow | /admin/ |
Disallow | /grid/ |
Disallow | /api/users/sign_in |
Disallow | /api/ |
Other Records
Field | Value |
---|---|
crawl-delay | 0.3 |
Other Records
Field | Value |
---|---|
sitemap | https://www.the-village.ru/api/sitemaps/sitemap.xml |
Warnings
- `clean-param` is not a known field.
- `host` is not a known field.