woolworths.com.au
robots.txt
Robots Exclusion Standard data for woolworths.com.au
Resource Scan
Scan Details
| Site Domain | woolworths.com.au |
| Base Domain | woolworths.com.au |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-09-03T02:53:20+00:00 |
| Next Scan | 2024-12-02T02:53:20+00:00 |
Last Successful Scan
| Scanned | 2024-05-06T22:29:37+00:00 |
| URL | https://woolworths.com.au/robots.txt |
| Redirect | https://www.woolworths.com.au/robots.txt |
| Redirect Domain | www.woolworths.com.au |
| Redirect Base | woolworths.com.au |
| Domain IPs | 23.52.113.39 |
| Redirect IPs | 23.52.113.39 |
| Response IP | 23.54.57.90 |
| Found | Yes |
| Hash | a04487fa2a0d4a2b1b3ccfb5c76cfb6cacd48577ac5690dfa45e017d86515e8c |
| SimHash | 8155c156cfbd |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /shop/checkout |
| Disallow | /checkout |
| Disallow | /shop/confirmation |
| Disallow | /shop/securelogin |
| Disallow | /Logs |
| Disallow | /shop/productgroup |
| Disallow | /shop/page/aem-homepage-test |
| Disallow | /shop/productdetails/*?*googleshop= |
| Disallow | /shop/productdetails/*?*region_id= |
| Disallow | /shop/productdetails/__partial/* |
| Disallow | /shop/search |
| Disallow | /shop/printrecipe |
| Disallow | *cddx* |
| Disallow | *icmpid* |
| Disallow | /shop/shopping/*?*region_id= |
| Disallow | /shop/__partial |
| Disallow | /shop/mylists |
| Disallow | /shop/myaccount |
| Disallow | /shop/discover/test-pages |
| Disallow | /shop/content-playbook* |
| Disallow | /Shop/DynamicContent2Panel* |
| Disallow | /shop/page/preview-homepage |
| Disallow | /shop/terms/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.woolworths.com.au/sitemap_index.xml |