asergeev.com
robots.txt
Robots Exclusion Standard data for asergeev.com
Resource Scan
Scan Details
| Site Domain | asergeev.com |
| Base Domain | asergeev.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-11-04T01:55:37+00:00 |
| Next Scan | 2026-02-02T01:55:37+00:00 |
Last Successful Scan
| Scanned | 2025-06-13T03:50:44+00:00 |
| URL | https://www.asergeev.com/robots.txt |
| Domain IPs | 104.21.66.22, 172.67.155.81, 2606:4700:3032::6815:4216, 2606:4700:3036::ac43:9b51 |
| Response IP | 172.67.155.81 |
| Found | Yes |
| Hash | 4bd66ff15174a8687effaec41c0064b5330ed6ebb29d275883d29bcff83391c9 |
| SimHash | e70868c7c6b7 |
Groups
*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /php/ |
| Disallow | /system/ |
| Disallow | /temp/apartment |
| Disallow | /temp/russia-mushrooms |
| Disallow | /webstat/ |
| Disallow | /files/soi/ |
| Disallow | /files/bookms/ |
| Disallow | /files/qatar/openfoam/run/ |
| Disallow | /files/qatar/openfoam/applications/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.asergeev.com/sitemapindex.xml |