joblist.com
robots.txt
Robots Exclusion Standard data for joblist.com
Resource Scan
Scan Details
| Site Domain | joblist.com |
| Base Domain | joblist.com |
| Scan Status | Ok |
| Last Scan | 2024-11-15T09:28:46+00:00 |
| Next Scan | 2024-11-22T09:28:46+00:00 |
Last Scan
| Scanned | 2024-11-15T09:28:46+00:00 |
| URL | https://joblist.com/robots.txt |
| Domain IPs | 104.22.50.181, 104.22.51.181, 172.67.38.29, 2606:4700:10::6816:32b5, 2606:4700:10::6816:33b5, 2606:4700:10::ac43:261d |
| Response IP | 172.67.38.29 |
| Found | Yes |
| Hash | ae24b6f4b32c18b85505591ed70aa49ef38fe9b0f9ec10a595ad45dd9a9ada78 |
| SimHash | e61c0e5ac291 |
Groups
*
| Rule | Path |
|---|---|
| Allow | /$ |
| Disallow | /search?* |
| Disallow | /uk/search?* |
| Disallow | /quiz?* |
| Disallow | /explore?* |
| Disallow | /c/* |
| Disallow | /clk* |
| Disallow | /framedListing* |
| Disallow | /b/individual-state* |
| Disallow | /b/location-job-title* |
| Disallow | /b/individual-location* |
| Disallow | /b/individual-job-title* |
| Disallow | /b/individual-industry* |
| Disallow | /b/location-industry* |
| Disallow | /b/state-industry* |
| Disallow | /b/state-job-title* |
| Disallow | /individual-job-title* |
| Disallow | /verify-email* |
| Disallow | /post-now |
| Disallow | /e/* |
| Disallow | /work-at-joblist |
| Disallow | /post |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.joblist.com/browse-sitemap-index.xml |
| sitemap | https://www.joblist.com/uk/browse-sitemap-index.xml |
| sitemap | https://www.joblist.com/ca/browse-sitemap-index.xml |
| sitemap | https://www.joblist.com/au/browse-sitemap-index.xml |
Warnings
- `​crawl-delay` is not a known field.
Comments