afr.com.au
robots.txt
Robots Exclusion Standard data for afr.com.au
Resource Scan
Scan Details
| Site Domain | afr.com.au |
| Base Domain | afr.com.au |
| Scan Status | Ok |
| Last Scan | 2024-05-02T17:29:33+00:00 |
| Next Scan | 2024-06-01T17:29:33+00:00 |
Last Scan
| Scanned | 2024-05-02T17:29:33+00:00 |
| URL | https://afr.com.au/robots.txt |
| Redirect | https://www.afr.com/robots.txt |
| Redirect Domain | www.afr.com |
| Redirect Base | afr.com |
| Domain IPs | 13.248.160.137, 75.2.43.150, 76.223.34.124, 99.83.186.106 |
| Redirect IPs | 151.101.130.133, 151.101.194.133, 151.101.2.133, 151.101.66.133, 2a04:4e42:200::645, 2a04:4e42:400::645, 2a04:4e42:600::645, 2a04:4e42::645 |
| Response IP | 199.232.46.133 |
| Found | Yes |
| Hash | 88a2f38df2f6ab18ed8602a008b9751a74123dda06ea339f1297c6d1bd9e9ff0 |
| SimHash | 5118d7e363f5 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /afr-newsletter-btb |
| Disallow | /afr-newsletter-mw5 |
| Disallow | /btb-newsletter-v1 |
| Disallow | /btb-newsletter-v2 |
| Disallow | /btb-newsletter-v3 |
| Disallow | /lifestyle-newsletter-v1 |
| Disallow | /lifestyle-newsletter-v2 |
| Disallow | /mw5-newsletter-v1 |
| Disallow | /mw5-newsletter-v2 |
| Disallow | /newsfeed |
| Disallow | /newsletter |
| Disallow | /search$ |
| Disallow | /search? |
| Disallow | /smi-newsletter |
| Disallow | /street-talk-early-look |
| Disallow | /the-election-brief |
| Disallow | /word-from-the-editor-v1 |
| Disallow | /word-from-the-editor-v2 |
| Disallow | /webview |