watchdust.com
robots.txt
Robots Exclusion Standard data for watchdust.com
Resource Scan
Scan Details
| Site Domain | watchdust.com |
| Base Domain | watchdust.com |
| Scan Status | Ok |
| Last Scan | 2024-11-09T01:55:59+00:00 |
| Next Scan | 2024-11-16T01:55:59+00:00 |
Last Scan
| Scanned | 2024-11-09T01:55:59+00:00 |
| URL | https://watchdust.com/robots.txt |
| Domain IPs | 104.21.68.154, 172.67.196.165, 2606:4700:3035::6815:449a, 2606:4700:3035::ac43:c4a5 |
| Response IP | 172.67.196.165 |
| Found | Yes |
| Hash | 88c4f8ad9221f2f7eb1e54c0c67ce9362c2249bc11f8ee2f1712f11a4c750eec |
| SimHash | 60386225ab37 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /anthology/ |
| Disallow | /blog/ |
| Disallow | /dust-anthology/ |
| Disallow | /video/ |
| Disallow | /elements/subscribe-pop-up/ |
| Disallow | /4k-new-batch/ |
| Disallow | /originals-slate/ |
| Disallow | /top-short-films/ |
| Allow | / |