williamsbrewing.com
robots.txt

Robots Exclusion Standard data for williamsbrewing.com

Resource Scan

Scan Details

Site Domain williamsbrewing.com
Base Domain williamsbrewing.com
Scan Status Ok
Last Scan4/3/2025, 10:24:42 AM
Next Scan 5/3/2025, 10:24:42 AM

Last Scan

Scanned4/3/2025, 10:24:42 AM
URL https://williamsbrewing.com/robots.txt
Domain IPs 104.21.52.207, 172.67.203.229, 2606:4700:3030::6815:34cf, 2606:4700:3037::ac43:cbe5
Response IP 104.21.52.207
Found Yes
Hash 7ef42cc8a3e917650bee8559735e843dca0b46fe281edad0c78f0dcef724adb5
SimHash 6151d033d215

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

msnbot

Rule Path
Disallow
Disallow /Admin/
Disallow /Checkout/
Disallow /Install/
Disallow /Members/
Disallow /Product/BuyProductDialog
Disallow /*.axd$

Other Records

Field Value
crawl-delay 20