boxd.it
robots.txt
Robots Exclusion Standard data for boxd.it
Resource Scan
Scan Details
Site Domain | boxd.it |
Base Domain | boxd.it |
Scan Status | Ok |
Last Scan | 2024-05-25T08:23:25+00:00 |
Next Scan | 2024-06-01T08:23:25+00:00 |
Last Scan
Scanned | 2024-05-25T08:23:25+00:00 |
URL | https://boxd.it/robots.txt |
Domain IPs | 104.21.16.167, 172.67.214.221, 2606:4700:3037::6815:10a7, 2606:4700:3037::ac43:d6dd |
Response IP | 104.21.16.167 |
Found | Yes |
Hash | aed87b549617830bad99984cf9ede81b98664eab8dde9db1c7d0eb74c59feb57 |
SimHash | ba55994345b2 |
Groups
*
Rule | Path | Comment |
---|---|---|
Disallow | /*/by/* | sorting options |
Disallow | /*/popular/this/* | popularity sorting options |
Disallow | /*/on/* | availability options |
Disallow | /*/tag/* | Members? tag lists |
Disallow | /*/genre/* | Films by genre |
Disallow | /*/country/* | Films by country |
Disallow | /*/language/* | Films by language |
Disallow | /*/decade/* | Films by decade |
Disallow | /films/year/* | Films by year |
Disallow | /films/*/year/* | Films by year |
Disallow | /films/*/size/large/* | Films with large posters (and therefore stats) |
Disallow | /*/friends/* | stuff grouped for users' friends |
Comments