colesliquorcorporate.com.au
robots.txt

Robots Exclusion Standard data for colesliquorcorporate.com.au

Resource Scan

Scan Details

Site Domain colesliquorcorporate.com.au
Base Domain colesliquorcorporate.com.au
Scan Status Ok
Last Scan2025-05-20T10:42:34+00:00
Next Scan 2025-06-19T10:42:34+00:00

Last Scan

Scanned2025-05-20T10:42:34+00:00
URL https://colesliquorcorporate.com.au/robots.txt
Redirect https://www.liquorconnection.com.au/robots.txt
Redirect Domain www.liquorconnection.com.au
Redirect Base liquorconnection.com.au
Domain IPs 207.120.40.165, 207.120.40.166, 207.120.40.167, 207.120.40.168, 207.120.40.169, 207.120.40.170
Redirect IPs 207.120.40.165, 207.120.40.166, 207.120.40.167, 207.120.40.168, 207.120.40.169, 207.120.40.170
Response IP 207.120.40.170
Found Yes
Hash 8ab1be7f8a033c1aef5bc9db322b1f82ac6fb6626a75dd710780bc4e9f96290c
SimHash 6a76da49cbb9

Groups

*

Rule Path
Disallow /*-and-*-range
Disallow /*-and-*-region
Disallow /*-and-*-wines
Disallow /*-and-*-spirits
Disallow /*-and-*-beer
Disallow /search
Disallow /api/order
Disallow /api/payment
Disallow /user
Disallow /api/*
Disallow /buy-again
Disallow /*fh_
Disallow /brands/*s%3D*

adsbot-google

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

screaming frog seo spider

Rule Path
Allow /

sitebulb

Rule Path
Allow /

mozilla/5.0 (compatible; botify; http://botify.com)

Rule Path
Allow /

mozilla/5.0 mobile (compatible; botify; http://botify.com)

Rule Path
Allow /

yakazbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

admantx

Rule Path
Disallow /

yandexrenderresourcesbot/1.0

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

snapchatads/1.0

Rule Path
Disallow /

snap url preview service; bot; snapchat

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

taboolabot/3.7

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

qwantify-prod/1.0

Rule Path
Disallow /

brightbot 1.0

Rule Path
Disallow /

pernod ricard - clicktobuy crawlerbot/1.0

Rule Path
Disallow /

claritybot

Rule Path
Disallow /149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/*

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

Comments

  • the below wildcard is intended to block bots crawling URLs that include the terms 'and' and various properties
  • These user-agents are fine
  • These user agents could cause unnecessary server requests