theaustralian.com.au
robots.txt
Robots Exclusion Standard data for theaustralian.com.au
Resource Scan
Scan Details
| Site Domain | theaustralian.com.au |
| Base Domain | theaustralian.com.au |
| Scan Status | Ok |
| Last Scan | 2024-09-21T02:39:56+00:00 |
| Next Scan | 2024-09-28T02:39:56+00:00 |
Last Scan
| Scanned | 2024-09-21T02:39:56+00:00 |
| URL | https://theaustralian.com.au/robots.txt |
| Redirect | https://www.theaustralian.com.au/robots.txt |
| Redirect Domain | www.theaustralian.com.au |
| Redirect Base | theaustralian.com.au |
| Domain IPs | 104.83.196.116 |
| Redirect IPs | 104.83.196.116 |
| Response IP | 23.207.180.112 |
| Found | Yes |
| Hash | e2693f1bb4c24fc6e078ba7f690a17bfafb901a697b49393837f0cba3a8bc0d9 |
| SimHash | 1114ce5aa892 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*/comments-* |
| Disallow | /*SIT* |
| Disallow | /*.swf |
| Disallow | /printpage/ |
| Disallow | /archive/ |
| Disallow | /tablet-t3/ |
| Disallow | /enewsletters/ |
| Disallow | /newsnetwork/ |
| Disallow | /news/*/collection/* |
| Disallow | /myaustralian/ |
| Disallow | /myaus/ |
| Disallow | /app-route/ |
| Disallow | /double-rainbow/ |
| Disallow | /it-test-only/ |
| Disallow | /business/stockhead/news/*/news-story/* |
| Disallow | /world/the-times/*/news-story/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.theaustralian.com.au/sitemap.xml |
| sitemap | https://www.theaustralian.com.au/news-sitemap.xml |
Comments