townswebarchiving.com
robots.txt
Robots Exclusion Standard data for townswebarchiving.com
Resource Scan
Scan Details
| Site Domain | townswebarchiving.com |
| Base Domain | townswebarchiving.com |
| Scan Status | Ok |
| Last Scan | 4/4/2025, 8:11:57 PM |
| Next Scan | 5/4/2025, 8:11:57 PM |
Last Scan
| Scanned | 4/4/2025, 8:11:57 PM |
| URL | https://townswebarchiving.com/robots.txt |
| Domain IPs | 104.21.72.168, 172.67.187.33 |
| Response IP | 104.21.72.168 |
| Found | Yes |
| Hash | 8522c821ca84f1e166dc71bffbec42c2f3399ff3d478274d4e7d9f48dc37f5b2 |
| SimHash | 5972817aaa9b |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /AA-DEL/ |
| Disallow | /anguline/ |
| Disallow | /bloomsbury/ |
| Disallow | /boe/ |
| Disallow | /boe_samples/ |
| Disallow | /borthwick/ |
| Disallow | /cgi-bin/ |
| Disallow | /clackmannanshire/ |
| Disallow | /clarks/ |
| Disallow | /client_files/ |
| Disallow | /constantcontact/ |
| Disallow | /crm/ |
| Disallow | /crm2/ |
| Disallow | /customer_uploads/ |
| Disallow | /dignity/ |
| Disallow | /edlc/ |
| Disallow | /feedbackform/ |
| Disallow | /FTP_ACCOUNTS/ |
| Disallow | /FTPACCOUNTS/ |
| Disallow | /ftpuser1/ |
| Disallow | /ftpuser2/ |
| Disallow | /goodward/ |
| Disallow | /images/ |
| Disallow | /jack/ |
| Disallow | /kingston/ |
| Disallow | /leicsuni/ |
| Disallow | /monica/ |
| Disallow | /movie_directory/ |
| Disallow | /newsletters/ |
| Disallow | /old_list/ |
| Disallow | /presentation/ |
| Disallow | /product-photography/ |
| Disallow | /project-pier/ |
| Disallow | /project-pier-bck/ |
| Disallow | /ramc/ |
| Disallow | /royalcollection/ |
| Disallow | /sedbergh/ |
| Disallow | /sites/ |
| Disallow | /spennymoor/ |
| Disallow | /SpennymoorSearch/ |
| Disallow | /st_demonstrations/ |
| Disallow | /subdomains/ |
| Disallow | /teesdale/ |
| Disallow | /LoughtonSearch/ |