Log In Sign Up

wine-searcher.com
robots.txt

Robots Exclusion Standard data for wine-searcher.com

Resource Scan

Scan Details

Site Domain wine-searcher.com
Base Domain wine-searcher.com
Scan Status Ok
Last Scan2024-05-26T14:51:57+00:00
Next Scan 2024-06-02T14:51:57+00:00

Last Scan

Scanned2024-05-26T14:51:57+00:00
URL https://wine-searcher.com/robots.txt
Redirect https://www.wine-searcher.com/robots.txt
Redirect Domain www.wine-searcher.com
Redirect Base wine-searcher.com
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 151.101.65.91
Found Yes
Hash 9c48fd009b6a7538fed0c05651d8413721f00236d3bf26d4619c2b54855bd219
SimHash 32bd65409d51

Groups

twitterbot

Rule Path
Disallow

*

Rule Path
Disallow /redirect*
Disallow /app-html/
Disallow /json-country/
Disallow /json-region/
Disallow /json-recommend/
Disallow /widgets/
Disallow /securimage/
Disallow /scratch/
Disallow /portal/
Disallow /site-search*
Disallow /wine-notes-search*
Disallow /sign-in*
Disallow /prof/index*
Disallow *pro_redirect*

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

ws nutch/solr/nutch-1.9

Rule Path
Disallow /

riddler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

exabot

Rule Path
Disallow /

snoothbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

daum

Rule Path
Disallow /

amazonec2

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

a6-indexer/1.0

Rule Path
Disallow /

wesee

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

ng/2.0

Rule Path
Disallow /

crystalsemantics

Rule Path
Disallow /

f-six

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

scrapybot

Rule Path
Disallow /

vocusbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ezooms.bot

Rule Path
Disallow /

skimlinks.com

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

wikiwix-bot

Rule Path
Disallow /

semantissimo

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

lemonsources

Rule Path
Disallow /

discobot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

ltbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

language-tools.com

Rule Path
Disallow /

trec-kba-bot

Rule Path
Disallow /

checkprivacy.or.kr

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

socialradarbot

Rule Path
Disallow /

havij

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mireobot

Rule Path
Disallow /

brandwatch

Rule Path
Disallow /

seoprofiler

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

bot clicker

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

exaleadcloudview

Rule Path
Disallow /

admedia

Rule Path
Disallow /

grepnetstat.com

Rule Path
Disallow /

genieo

Rule Path
Disallow /

nerdbynature

Rule Path
Disallow /

genieo

Rule Path
Disallow /

sindup

Rule Path
Disallow /

netseer

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

netseer.com

Rule Path
Disallow /

nagios

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

ning/1.0

Rule Path
Disallow /

feedretriever

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

argus presse

Rule Path
Disallow /

ibm ica crawler

Rule Path
Disallow /

nutch agent

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

linkscrawler

Rule Path
Disallow /

archive.org

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

netestate

Rule Path
Disallow /

mynutchtest

Rule Path
Disallow /

admantx

Rule Path
Disallow /

genieo.com

Rule Path
Disallow /

mixdata.com

Rule Path
Disallow /

www.wise-guys.nl

Rule Path
Disallow /

www.wesee.com

Rule Path
Disallow /

www.200please.com

Rule Path
Disallow /

ubermetrics-technologies

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

wasalive

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /
Back to top