bofrost.fr
robots.txt
Robots Exclusion Standard data for bofrost.fr
Resource Scan
Scan Details
Site Domain | bofrost.fr |
Base Domain | bofrost.fr |
Scan Status | Failed |
Failure Stage | Fetching resource. |
Failure Reason | Server returned a client error. |
Last Scan | 2024-10-14T14:50:33+00:00 |
Next Scan | 2025-01-12T14:50:33+00:00 |
Last Successful Scan
Scanned | 2024-03-11T14:28:15+00:00 |
URL | https://bofrost.fr/robots.txt |
Redirect | https://www.bofrost.fr/robots.txt |
Redirect Domain | www.bofrost.fr |
Redirect Base | bofrost.fr |
Domain IPs | 52.174.254.89 |
Redirect IPs | 23.215.7.23, 23.215.7.25, 2600:1413:b000:1b::17d7:717, 2600:1413:b000:1b::17d7:719 |
Response IP | 23.52.40.32 |
Found | Yes |
Hash | e9f86f57c64f0b799f3f3bbf075ef101655758148e07e46d443dc7e4e94c2811 |
SimHash | ec525f1eedfc |
Groups
*
Rule | Path |
---|---|
Disallow | /cart |
Disallow | /checkout |
Disallow | /account |
Disallow | /shop/ajax/ |
Other Records
Field | Value | Comment |
---|---|---|
crawl-delay | 10 | 10 seconds between page requests |
Other Records
Field | Value |
---|---|
sitemap | https://www.bofrost.fr/sitemap.xml |
Warnings
- `request-rate` is not a known field.
- `visit-time` is not a known field.
Comments