classifieds.theprovince.com
robots.txt
Robots Exclusion Standard data for classifieds.theprovince.com
Resource Scan
Scan Details
Site Domain | classifieds.theprovince.com |
Base Domain | theprovince.com |
Scan Status | Ok |
Last Scan | 2024-06-04T22:04:24+00:00 |
Next Scan | 2024-07-04T22:04:24+00:00 |
Last Scan
Scanned | 2024-06-04T22:04:24+00:00 |
URL | https://classifieds.theprovince.com/robots.txt |
Domain IPs | 52.33.82.82, 52.39.84.17, 54.70.206.26 |
Response IP | 52.39.84.17 |
Found | Yes |
Hash | fda73e11b809729011a31128d426b031737e78037a0151e879bafcb4ffa738a7 |
SimHash | 6b0537730be7 |
Groups
*
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
*
Rule | Path |
---|---|
Disallow | /*?*search_type* |
*
Rule | Path |
---|---|
Disallow | /search* |
*
Rule | Path |
---|---|
Disallow | /*?*ap_search_* |
*
Rule | Path |
---|---|
Disallow | /ajax/post_form/ |
*
Rule | Path |
---|---|
Disallow | /admin/ |
*
Rule | Path |
---|---|
Disallow | /*-admin/ |
*
Rule | Path |
---|---|
Disallow | /manage-*/ |
*
Rule | Path |
---|---|
Disallow | /create-*/ |
*
Rule | Path |
---|---|
Disallow | /edit-*/ |
*
Rule | Path |
---|---|
Disallow | /claim-*/ |
*
Rule | Path |
---|---|
Disallow | /wp-admin/ |
Allow | /wp-admin/admin-ajax.php |
Warnings
- 8 invalid lines.
Comments