justauto.com.au
robots.txt
Robots Exclusion Standard data for justauto.com.au
Resource Scan
Scan Details
| Site Domain | justauto.com.au |
| Base Domain | justauto.com.au |
| Scan Status | Ok |
| Last Scan | 2024-05-31T17:39:38+00:00 |
| Next Scan | 2024-06-07T17:39:38+00:00 |
Last Scan
| Scanned | 2024-05-31T17:39:38+00:00 |
| URL | https://justauto.com.au/robots.txt |
| Redirect | https://www.justauto.com.au/robots.txt |
| Redirect Domain | www.justauto.com.au |
| Redirect Base | justauto.com.au |
| Domain IPs | 104.18.10.35, 104.18.11.35, 2606:4700::6812:a23, 2606:4700::6812:b23 |
| Redirect IPs | 104.18.10.35, 104.18.11.35, 2606:4700::6812:a23, 2606:4700::6812:b23 |
| Response IP | 104.18.11.35 |
| Found | Yes |
| Hash | 76d209fb4f1e3e9abbfdd040cf311009c327e3f942d72acf89e6e39893532feb |
| SimHash | 45a284c80c90 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /dashboard/ |
| Disallow | /info/ |
| Disallow | /user/login/ |
| Disallow | /flag/flag/favourite_ad/* |
| Disallow | /flag/unflag/favourite_ad/* |
| Disallow | */show-number |
| Disallow | /node/ |
| Disallow | /node/add/ |
| Disallow | /subscribe/confirmation |
| Disallow | /webhook/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.justauto.com.au/sitemap.xml |
| sitemap | https://www.justauto.com.au/search_item/sitemap.xml |
| sitemap | https://www.justauto.com.au/taxonomy/sitemap.xml |
Comments