arbashjw.com
robots.txt
Robots Exclusion Standard data for arbashjw.com
Resource Scan
Scan Details
Site Domain | arbashjw.com |
Base Domain | arbashjw.com |
Scan Status | Failed |
Failure Stage | Fetching resource. |
Failure Reason | Server returned a client error. |
Last Scan | 2024-07-06T11:22:00+00:00 |
Next Scan | 2024-10-04T11:22:00+00:00 |
Last Successful Scan
Scanned | 2023-11-17T11:19:03+00:00 |
URL | https://arbashjw.com/robots.txt |
Domain IPs | 104.21.39.110, 172.67.144.147, 2606:4700:3032::6815:276e, 2606:4700:3032::ac43:9093 |
Response IP | 104.21.39.110 |
Found | Yes |
Hash | de3473cf74e607fac89ba3f71e8bcc10677739850be3c9e4562917840498f933 |
SimHash | 6307677101a7 |
Groups
*
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
Other Records
Field | Value |
---|---|
sitemap | https://arbashjw.com/sitemap.xml |
Warnings
- 10 invalid lines.