libwww.freelibrary.org
robots.txt
Robots Exclusion Standard data for libwww.freelibrary.org
Resource Scan
Scan Details
| Site Domain | libwww.freelibrary.org |
| Base Domain | freelibrary.org |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Request timed out. |
| Last Scan | 2024-08-13T01:48:38+00:00 |
| Next Scan | 2024-10-12T01:48:38+00:00 |
Last Successful Scan
| Scanned | 2024-05-23T01:40:40+00:00 |
| URL | https://libwww.freelibrary.org/robots.txt |
| Domain IPs | 38.98.224.22 |
| Response IP | 38.98.224.22 |
| Found | Yes |
| Hash | 60cb6037fd66419b56aeaa887f42969fe5bbf96c47f98dada159e969bc5aaa8e |
| SimHash | 5b22447c5f13 |
Groups
*
| Rule | Path |
|---|---|
| Allow | /followerossified.cfm |
| Disallow | /aspnet_client/ |
| Disallow | /CFIDE/ |
| Disallow | /digicol/ |
| Disallow | /elecres/ |
| Disallow | /explore/reviews/ |
| Disallow | /illiad/ |
| Disallow | /assets/images/ |
| Disallow | /include/ |
| Disallow | /misc/ |
| Disallow | /podcast/media/20130402-blaineh.mp3 |
| Disallow | /staffweb/ |