mall-net.com
robots.txt
Robots Exclusion Standard data for mall-net.com
Resource Scan
Scan Details
| Site Domain | mall-net.com |
| Base Domain | mall-net.com |
| Scan Status | Ok |
| Last Scan | 2025-10-17T12:12:40+00:00 |
| Next Scan | 2025-11-16T12:12:40+00:00 |
Last Scan
| Scanned | 2025-10-17T12:12:40+00:00 |
| URL | https://mall-net.com/robots.txt |
| Domain IPs | 70.39.144.94 |
| Response IP | 70.39.144.94 |
| Found | Yes |
| Hash | 9d58eae01add044f6330e3b757fe38f2b670344e89458efe35f3d11559ad0a4c |
| SimHash | 125459894752 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /jvv | Constant flux, client specific. |
| Disallow | /team-javilk | Constant flux, client specific. |
| Disallow | /bus | Constant flux, client specific. |
| Disallow | /webcons | transient data for specific clients |
| Disallow | /cgibin | confuses the heck out of robots and droids. |
| Disallow | /cgi-bin | confuses the heck out of robots and droids. |
| Disallow | /bioergo | Transient |
| Disallow | /bioc | - |
| Disallow | /medlib | books have been donated |
| Disallow | /tree | - |
| Disallow | /C30 | - |
| Disallow | /C35 | - |
| Disallow | /C40 | - |
| Disallow | /C43 | - |
| Disallow | /C44 | - |
| Disallow | /mcs/r | restricted data |
| Disallow | /mcs/porph | restricted data |
| Disallow | /javilk/r | transient |
| Disallow | /javilk/marmara | unfinished |
| Disallow | /class | - |
| Disallow | /camping | - |
| Disallow | /ops | - |
| Disallow | /2000/r | Just gets too complex for indexing |
| Disallow | /icon | Images |
| Disallow | /sunset | Images... no longer there. |
Comments