scarabeus.ru
robots.txt
Robots Exclusion Standard data for scarabeus.ru
Resource Scan
Scan Details
| Site Domain | scarabeus.ru |
| Base Domain | scarabeus.ru |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a server error. |
| Last Scan | 2024-09-19T20:38:58+00:00 |
| Next Scan | 2024-10-03T20:38:58+00:00 |
Last Successful Scan
| Scanned | 2024-09-10T09:19:11+00:00 |
| URL | https://scarabeus.ru/robots.txt |
| Domain IPs | 2a00:15f8:a000:5:1:11:4:3d70, 2a00:15f8:a000:5:1:12:4:3d70, 2a00:15f8:a000:5:1:13:4:3d70, 2a00:15f8:a000:5:1:14:4:3d70, 90.156.201.104, 90.156.201.113, 90.156.201.38, 90.156.201.46 |
| Response IP | 90.156.201.113 |
| Found | Yes |
| Hash | 8c3754c2f877e8785a29f5c074b2879a43a1b84acbda7d1ea8c8e7741a3a04d7 |
| SimHash | fd410c101db3 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /assets/cache/ |
| Disallow | /assets/docs/ |
| Disallow | /assets/export/ |
| Disallow | /assets/import/ |
| Disallow | /assets/modules/ |
| Disallow | /assets/plugins/ |
| Disallow | /assets/snippets/ |
| Disallow | /install/ |
| Disallow | /manager/ |
| Disallow | /? |
| Disallow | /*?id= |
Other Records
| Field | Value |
|---|---|
| sitemap | https://scarabeus.ru/sitemap.xml |
Warnings
- `host` is not a known field.