justcars.com.au
robots.txt
Robots Exclusion Standard data for justcars.com.au
Resource Scan
Scan Details
| Site Domain | justcars.com.au |
| Base Domain | justcars.com.au |
| Scan Status | Ok |
| Last Scan | 2024-11-16T02:26:38+00:00 |
| Next Scan | 2024-11-23T02:26:38+00:00 |
Last Scan
| Scanned | 2024-11-16T02:26:38+00:00 |
| URL | https://justcars.com.au/robots.txt |
| Redirect | https://www.justcars.com.au/robots.txt |
| Redirect Domain | www.justcars.com.au |
| Redirect Base | justcars.com.au |
| Domain IPs | 104.18.22.158, 104.18.23.158, 2606:4700::6812:169e, 2606:4700::6812:179e |
| Redirect IPs | 104.18.22.158, 104.18.23.158, 2606:4700::6812:169e, 2606:4700::6812:179e |
| Response IP | 104.18.22.158 |
| Found | Yes |
| Hash | 0a0f313e4c93420623f639d534e474176f5a16f568576484c756325725636e16 |
| SimHash | 478394cc049a |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /dashboard/ |
| Disallow | /info/ |
| Disallow | /user/login/ |
| Disallow | /flag/flag/favourite_ad/* |
| Disallow | /flag/unflag/favourite_ad/* |
| Disallow | */show-number |
| Disallow | /node/ |
| Disallow | /node/add/ |
| Disallow | /subscribe/confirmation |
| Disallow | /webhook/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.justcars.com.au/sitemap.xml |
| sitemap | https://www.justcars.com.au/categories/sitemap.xml |
| sitemap | https://www.justcars.com.au/business/sitemap.xml |
| sitemap | https://www.justcars.com.au/event/sitemap.xml |
| sitemap | https://www.justcars.com.au/article/sitemap.xml |
| sitemap | https://www.justcars.com.au/search_item/sitemap.xml |
| sitemap | https://www.justcars.com.au/taxonomy/sitemap.xml |
| sitemap | https://www.justcars.com.au/article_google_news/sitemap.xml |
Comments