woolworthsgroup.com.au
robots.txt

Robots Exclusion Standard data for woolworthsgroup.com.au

Resource Scan

Scan Details

Site Domain woolworthsgroup.com.au
Base Domain woolworthsgroup.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-15T01:51:49+00:00
Next Scan 2026-03-15T01:51:49+00:00

Last Successful Scan

Scanned2024-05-01T17:45:24+00:00
URL https://woolworthsgroup.com.au/robots.txt
Redirect https://www.woolworthsgroup.com.au/robots.txt
Redirect Domain www.woolworthsgroup.com.au
Redirect Base woolworthsgroup.com.au
Domain IPs 23.52.113.39
Redirect IPs 23.52.113.39
Response IP 23.54.57.90
Found Yes
Hash 54f7893b1ffc3dcd34805501eb0ce2bb416e8ecabe5e97af3d9e72c8c972786c
SimHash 410cc0a0ce93

Groups

*

Rule Path
Allow /

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://woolworthsgroup.com.au/sitemap.xml

Comments

  • Block Bad Bots