diskunion.net
robots.txt
Robots Exclusion Standard data for diskunion.net
Resource Scan
Scan Details
Site Domain | diskunion.net |
Base Domain | diskunion.net |
Scan Status | Ok |
Last Scan | 2024-09-08T01:59:36+00:00 |
Next Scan | 2024-10-08T01:59:36+00:00 |
Last Scan
Scanned | 2024-09-08T01:59:36+00:00 |
URL | https://diskunion.net/robots.txt |
Domain IPs | 52.175.144.86 |
Response IP | 52.175.144.86 |
Found | Yes |
Hash | 51d6b95605ebbd600443b7ca3d98f72bc75beeb47a41d77a37a54c7a0c398fc1 |
SimHash | 80101f1af2a2 |
Groups
*
Rule | Path |
---|---|
Disallow | /*/ct/sub/01/ |
Disallow | /*/ct/sub/02/ |
Disallow | /*/ct/sub/03/ |
Disallow | /*/ct/sub/04/ |
Disallow | /*/ct/sub/05/ |
Disallow | /*/ct/sub/06/ |
Disallow | /*/ct/sub/08/ |
Disallow | /*/ct/sub/09/ |
Disallow | /*/ct/sub/12/ |
Disallow | /*/ct/sub/13/ |
Disallow | /*/ct/sub/14/ |
Disallow | /*/ct/sub/15/ |
Disallow | /*/ct/sub/18/ |
Disallow | /*/ct/sub/20/ |
Disallow | /*/ct/sub/21/ |
Disallow | /*/ct/used_list/ |
Disallow | /util/ct/now |
Disallow | /st/in-house_paper/ |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
googlebot-mobile
Rule | Path |
---|---|
Disallow | /mobile/ec/ |
Disallow | /mobile/*/search |
Disallow | /*/ct/sub/01/ |
Disallow | /*/ct/sub/02/ |
Disallow | /*/ct/sub/03/ |
Disallow | /*/ct/sub/04/ |
Disallow | /*/ct/sub/05/ |
Disallow | /*/ct/sub/06/ |
Disallow | /*/ct/sub/08/ |
Disallow | /*/ct/sub/09/ |
Disallow | /*/ct/sub/12/ |
Disallow | /*/ct/sub/13/ |
Disallow | /*/ct/sub/14/ |
Disallow | /*/ct/sub/15/ |
Disallow | /*/ct/sub/18/ |
Disallow | /*/ct/sub/20/ |
Disallow | /*/ct/sub/21/ |
Disallow | /*/ct/used_list/ |
Disallow | /util/ct/now |
Warnings
- 2 invalid lines.