Log In Sign Up

witchery.com.au
robots.txt

Robots Exclusion Standard data for witchery.com.au

Resource Scan

Scan Details

Site Domain witchery.com.au
Base Domain witchery.com.au
Scan Status Ok
Last Scan2025-11-28T01:19:15+00:00
Next Scan 2025-12-05T01:19:15+00:00

Last Scan

Scanned2025-11-28T01:19:15+00:00
URL https://witchery.com.au/robots.txt
Redirect https://www.witchery.com.au/robots.txt
Redirect Domain www.witchery.com.au
Redirect Base witchery.com.au
Domain IPs 76.76.21.21
Redirect IPs 104.18.33.243, 172.64.154.13, 2606:4700:4405::ac40:9a0d, 2a06:98c1:3100::6812:21f3
Response IP 172.64.154.13
Found Yes
Hash ae447b86c5d2890c1ea65bc5b1843de915d3c1f20d3fcf3a569dbf949c50e863
SimHash 6d14dd26f2e1

Groups

*

Rule Path
Allow /
Disallow /*Z%3D*
Disallow /*z%3D*
Disallow /*5B*
Disallow /*%5B*
Disallow /*q%3D*
Disallow /*action*
Disallow /*bvstate*
Disallow /*catid%3D*
Disallow /cart*
Disallow /checkout*
Disallow /complete*
Disallow /payment*
Disallow /CR_DigitalWallet.ashx

adsbot-google

Rule Path
Disallow /default.aspx*

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.8
Back to top

Comments

  • AdsBot-Google has to be blocked explicitly (https://support.google.com/adwords/answer/2404197?hl=en)
  • Pinterest Bot Rate Limit
  • Pinterest - Bot Rate Limit - Format from (https://help.pinterest.com/en-gb/business/article/pinterest-crawler)
Back to top

Warnings

  • `crawl delay` is not a known field.
  • `user agent` is not a known field.
Back to top