cwi-portal.com
robots.txt
Robots Exclusion Standard data for cwi-portal.com
Resource Scan
Scan Details
| Site Domain | cwi-portal.com |
| Base Domain | cwi-portal.com |
| Scan Status | Ok |
| Last Scan | 2025-09-21T00:09:31+00:00 |
| Next Scan | 2025-10-05T00:09:31+00:00 |
Last Scan
| Scanned | 2025-09-21T00:09:31+00:00 |
| URL | https://www.cwi-portal.com/robots.txt |
| Domain IPs | 23.59.168.115 |
| Response IP | 23.54.118.49 |
| Found | Yes |
| Hash | c14f7229084dcd056363a23acca25405c12b6558705ceaa5e1b4183aef49f3d8 |
| SimHash | e832d9bb42f4 |
Groups
*
| Product | Comment |
|---|---|
| * | any robot |
| Rule | Path |
|---|---|
| Disallow | /user/ |
| Disallow | /settings/ |
| Disallow | /account/ |
| Disallow | /payment/ |
| Disallow | /paypal/ |
| Disallow | /signup/submission/* |
| Disallow | /signup/resume/* |
| Disallow | /Account/ |
| Disallow | /Publisher/StaffAccept/ |
| Disallow | /StaffAccept/ |
| Disallow | /SubmissionEdit/ |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 10 | number of seconds to wait between subsequent visits |
Comments