scrapo.co
robots.txt
Robots Exclusion Standard data for scrapo.co
Resource Scan
Scan Details
| Site Domain | scrapo.co |
| Base Domain | scrapo.co |
| Scan Status | Ok |
| Last Scan | 2024-09-09T11:09:57+00:00 |
| Next Scan | 2024-10-09T11:09:57+00:00 |
Last Scan
| Scanned | 2024-09-09T11:09:57+00:00 |
| URL | https://scrapo.co/robots.txt |
| Domain IPs | 13.33.88.127, 13.33.88.128, 13.33.88.28, 13.33.88.36 |
| Response IP | 13.33.88.127 |
| Found | Yes |
| Hash | 4c7c6be4a4612f7756bcf3180be4064978e8924f456fe6a39740a1991ddf9722 |
| SimHash | e405fc0ac4be |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /error/ |
| Disallow | /site/error/ |
| Disallow | /profile/update |
| Disallow | /checklist/ |
| Disallow | /profile/ |
| Disallow | /profile/followings |
| Disallow | /profile/followers |
| Disallow | /site/logout/ |
| Disallow | /site/login/ |
| Disallow | /login/ |
| Disallow | /zupaadmin/ |
| Disallow | /deploy/ |
| Disallow | /product/product/search? |
| Disallow | /checklist |
| Disallow | /store-settings |
| Disallow | /payments |
| Disallow | /purchase-orders |
| Disallow | /bargain-history |
| Disallow | /product/create |
| Disallow | /stream |
| Disallow | /messages |
| Disallow | /invitations |
| Disallow | /notifications |
| Disallow | /map/users |
| Disallow | /index?ProductSearch |
Other Records
| Field | Value |
|---|---|
| sitemap | https://scrapo.com/sitemap.xml |
Warnings
- `host` is not a known field.