crewmarket.net
robots.txt
Robots Exclusion Standard data for crewmarket.net
Resource Scan
Scan Details
| Site Domain | crewmarket.net |
| Base Domain | crewmarket.net |
| Scan Status | Ok |
| Last Scan | 2026-03-07T21:52:19+00:00 |
| Next Scan | 2026-03-14T21:52:19+00:00 |
Last Scan
| Scanned | 2026-03-07T21:52:19+00:00 |
| URL | http://crewmarket.net/robots.txt |
| Domain IPs | 77.87.192.196 |
| Response IP | 77.87.192.196 |
| Found | Yes |
| Hash | a2fd08460974fb3706935768b6583f1b6cadd55c10ba17b286ef2c7bd3b3c7cf |
| SimHash | 6b049c48ce90 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /jobs/search/ |
| Disallow | /jobs/advanced-search/ |
| Disallow | /jobs/type/ |
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
| Disallow | /wp-content/ |
| Disallow | /*?* |
| Allow | /*.png$ |
| Allow | /*.css$ |
| Allow | /*.js$ |
| Allow | /wp-content/uploads/application_form.xls |
| Allow | /board/$ |
| Disallow | /board/* |
| Disallow | /contact/ |
| Disallow | /reklama/ |
| Disallow | /services/ |
| Disallow | /button/ |
| Disallow | /english/ |
| Disallow | /ask-jurist/ |
| Disallow | /law/ |
| Disallow | /jobs/add/$ |
Other Records
| Field | Value |
|---|---|
| sitemap | http://crewmarket.net/sitemap.xml |
Warnings
- `host` is not a known field.