wine.com
robots.txt

Robots Exclusion Standard data for wine.com

Resource Scan

Scan Details

Site Domain wine.com
Base Domain wine.com
Scan Status Ok
Last Scan2024-05-26T00:39:40+00:00
Next Scan 2024-06-09T00:39:40+00:00

Last Scan

Scanned2024-05-26T00:39:40+00:00
URL https://wine.com/robots.txt
Redirect https://www.wine.com/robots.txt
Redirect Domain www.wine.com
Redirect Base wine.com
Domain IPs 18.238.171.4, 18.238.171.79, 18.238.171.8, 18.238.171.90
Redirect IPs 151.101.130.137, 151.101.194.137, 151.101.2.137, 151.101.66.137, 2a04:4e42:200::649, 2a04:4e42:400::649, 2a04:4e42:600::649, 2a04:4e42::649
Response IP 199.232.46.137
Found Yes
Hash 53662665bad50b52bfe2aaf1a05a320e539cbc6122a5dcf84462256905f2a48f
SimHash eb76dc68c0a3

Groups

*

Rule Path
Disallow /account
Disallow /search
Disallow /activity
Disallow /checkout
Disallow /cart
Disallow /winecom
Disallow /api
Disallow /*?*showOutOfStock=*
Disallow /*?*sortBy=*
Disallow /*?*ratingmax=*
Disallow /*?*communityratingmin=*
Disallow /*?*communityratingmax=*
Disallow /*?*abvmin=*
Disallow /*?*abvmax=*
Disallow /*?*showRegionMap=*
Disallow /*7155-*-*-*-*
Disallow /*7157-*-*-*-*
Disallow /*7151-*-*-*-*

mj12bot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

sosospider

Rule Path
Disallow /

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

zing-bottabot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 90

zing-bottabot/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 90

Other Records

Field Value
sitemap https://www.wine.com/sitemap.xml

Warnings

  • `noindex` is not a known field.