wineweb.com
robots.txt

Groups

a .net web crawler
a1 website download/1.* (*) miggibot
abot/*
acadiauniversitywebcensusclient
activerefresh*
ad muncher*
aiderss/2.0 (aiderss.com)
amico alpha * (*) gecko/* amicoalpha/*
androiddownloadmanager
annotate_google; http://ponderer.org/*
anonymisiert*
anonymizer/*
anonymizied*
anonymous*
anonymous/*
ahrefsbot
artera (version *)
atomic_email
atomic_email_hunter/*
autohotkey
automate5
b2w/*
backstreet browser *
baidu
baiduspider
basichttp/*
bdfetch
beamer*
bilgibot/*
bitbeamer/*
bittorrent/*
blocknote.net
bluecoat proxysg
bot/* (bot; *bot@bot.bot)
busiversebot/v1.0 (http://www.busiverse.com/bot.php)
camcrawler*
cast
cazoodlebot/*
ce-preload
cerberiandrtrs/*
cfnetwork/*
cfschedule*
cherrypicker*/*
chilkat/*
cms crawler (?http://buytaert.net/crawler/)
cobweb/*
cocoal.icio.us/* (*)*
coldfusion*
contactbot/*
copyright sheriff (*)
copyrightcheck*
crawl_application
cterm/*
curl*
custo*
cyberpatrol*
cydralspider/*
cz32ts
da *
datacha0s/*
datafountains/dmoz downloader*
deepindexer*
der gro\xdfe bildersauger*
desktop sidebar*
disco pump *
domainsbotbot/1.*
dotbot
download demon*
download express*
download master*
download ninja*
download wonder*
downloadsession*
e-societyrobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
easydl/*
ecatch*
emailcollector*
emailsearcher
emailsiphon*
emailwolf*
envolk/* (?http://www.envolk.com/envolk*)
envolk?its?spider/* (?http://www.envolk.com/envolk*)
epsilon softworks' mailmunky
estylesearch * (compatible; msie 6.0; windows nt 5.0)
exabot-images/1.0
exabot-test/*
exabot/2.0
exabot/3.0
exactseek-pagereaper-* (crawler@exactseek.com)
exalead ng/*
extractorpro*
extreme picture finder
ezic.com http agent *
fairad client*
fangcrawl/*
favorstarbot/*
fdm 1.x
feed::find/*
feedfetcher-google*
feedfetcher-google-igooglegadgets*
fetch libfetch/*
fget*
findfiles.net/* (robot;test_robot@gmx-topmail.de)
flaming attackbot*
flashget
flatarts_favico
followsite.com (*)
foobot*
fooky.com/scorpionbot/scoutout;*
forschungsportal/*
fotochecker
franklin locator*
freshdownload/*
fyberspider*
gamespyhttp/*
getright/*
getrightpro/*
getsmart/*
gnome-vfs/*
go!zilla*
go-ahead-got-it*
gozilla/*
gsa-crawler*
gulper web *
gurujibot/1.*
harvest/*
hatena antenna/*
hatena bookmark/*
hatena rss/*
hatena::crawler/*
hatenascreenshot*
hcat/*
healthbot/health_and_longevity_project_(healthhaven.com)
hiddenmarket-*
hitcrawler_0.*
hloader
holmes/*
hoowwwer/*
html2jpg blackbox, http://www.html2jpg.com
htmlparser/*
http generic
http://anonymouse.org/*
http://arachnode.net*
http://hilfe.acont.de/bot.html acontbot
httpclient*
httperf/*
httpfetch/*
httpgrab
httpsession
httpunit/*
hyperestraier/*
ia_archiver*
ice_getfile
iconsurf/2.*
icopyright conductor*
ie/6.01 (cp/m; 8-bit*)
iexplore.exe
igetter/*
inet - eureka app
inetbot/* (?http://www.inetbot.com/bot.html)
ineturl/*
ineturl:/*
infociousbot (?http://corp.infocious.com/tech_crawler.php)
inne: mozilla/4.0 (compatible; cerberian drtrs*)
internet exploiter/*
internet explore *
internet explorer *
internet ninja*
internetarchive/*
ip*works!*/*
ipiumbot laurion(dot)com
irlbot/*
irssiurllog/*
iwagent/*
jetbrains omea reader*
jpluck/*
just-crawler(*)
kapere (http://www.kapere.com)
kbeebot/0.*
kevin http://*
kolinka forum search (www.kolinka.com)
kontiki client*
kretrieve/
lachesis
leechftp
leechget*
letscrawl.com/1.0*
lftp/3.2.1
libcurl-agent/*
libweb/clshttp*
liferea/1.* (linux; *; http://liferea.sf.net/)
lightningdownload/*
lincoln state web browser
link valet online*
linkextractorpro*
links4us-crawler,*
lmqueuebot/*
looq/0.1*
lorkyll *.* -- lorkyll@444.net
lsearch/sondeur
lucidmedia clicksense/4.?
lwp*
made by zmeu @ whitehat v0.* (www.whitehat.ro)
mapoftheinternet.com?(?http://mapoftheinternet.com)
metaproducts download express/*
metatagsdir/*
mfc foundation class library*
mfc_tear_sample
mfhttpscan
microsoft bits/*
microsoft data access internet publishing provider cache manager
microsoft data access internet publishing provider dav*
microsoft data access internet publishing provider protocol discovery
microsoft internet explorer
microsoft office existence discovery
microsoft office protocol discovery
microsoft office/* (*picture manager*)
microsoft url control*
microsoft visio msie
microsoft windows network diagnostics
microsoft-webdav-miniredir/*
missigua locator*
mister pix*
mono browser capabilities updater*
moozilla
morfeus fucking scanner
movabletype/*
mozilla/* (compatible; linktiger/*; *http://www.linktiger.com*)
mozilla/* (compatible; offbyone; windows*) webster pro v3.*
mozilla/* (turingos; turing machine; 0.0)
mozilla/0.9* no dos :) (linux*)
mozilla/2.0 (compatible; newt activex; win32)
mozilla/3.0 (compatible; indy library)
mozilla/4.0 (compatible; advanced email extractor*)
mozilla/4.0 (compatible; bordermanager*)
mozilla/4.0 (compatible; botw spider; *http://botw.org)
mozilla/4.0 (compatible; cerberian drtrs*)
mozilla/4.0 (compatible; getleft*)
mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
mozilla/4.0 (compatible; msie ?.0; safersurf*)
mozilla/4.0 (compatible; msie 4.01; vonna.com b o t)
mozilla/4.0 (compatible; msie 6.0; bluecoat drtr)
mozilla/4.0 (compatible; scumbot/*; linux/*)
mozilla/4.0 (compatible; spider; linux)
mozilla/4.0 (compatible; trend micro tmdr 1.*
mozilla/4.0 (compatible; win32)
mozilla/5.0 (*) gecko/* firefox/2.0 oneriot/1.0 (http://www.oneriot.com)
mozilla/5.0 (*) voilabot*
mozilla/5.0 (*http://gnomit.com/) gecko/* gnomit/1.0
mozilla/5.0 (compatible; aboutusbot/*)
mozilla/5.0 (compatible; archive.org_bot*)
mozilla/5.0 (compatible; buzzrankingbot/*)
mozilla/5.0 (compatible; charlotte/*; *)
mozilla/5.0 (compatible; clixsense; http://www.clixsense.com/)
mozilla/5.0 (compatible; crawly/1.*; +http://*/crawler.html)
mozilla/5.0 (compatible; del.icio.us-thumbnails/*; *) khtml/* (like gecko)
mozilla/5.0 (compatible; dkimrepbot/*)
mozilla/5.0 (compatible; dotbot/*; http://www.dotnetdotcom.org/*)
mozilla/5.0 (compatible; exabot-images/3.0*)
mozilla/5.0 (compatible; exabot/3.0*)
mozilla/5.0 (compatible; ipcheck server monitor*)
mozilla/5.0 (compatible; jadynavebot; *http://www.jadynave.com/robot*
mozilla/5.0 (compatible; kaloogabot; http://www.kalooga.com/info.html?page=crawler)
mozilla/5.0 (compatible; legalanalysisagent/1.*; http://www.legalx.net)
mozilla/5.0 (compatible; mj12bot/v1.*)
mozilla/5.0 (compatible; netcraftsurveyagent/1.0; *info@netcraft.com)
mozilla/5.0 (compatible; nextthing.org/*)
mozilla/5.0 (compatible; ngbot/*)
mozilla/5.0 (compatible; oso;*
mozilla/5.0 (compatible; scoutjet; +http://www.scoutjet.com/)
mozilla/5.0 (compatible; scrubby/*; +http://www.scrubtheweb.com/abs/meta-check.html)
mozilla/5.0 (compatible; seznam screenshot-generator 2.0;*)
mozilla/5.0 (compatible; speedy spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
mozilla/5.0 (compatible; theophrastus/*)
mozilla/5.0 (compatible; twitturls; +http://twitturls.com)
mozilla/5.0 (compatible; viralheat bot/*)
mozilla/5.0 (compatible; webbot/*)
mozilla/5.0 (compatible; webscan v0.*; +http://otc.dyndns.org/webscan/)
mozilla/5.0 (compatible; yodaobot/1.*)
mozilla/5.0 (compatible;yodaobot-image/1.*)
mozilla/5.0 (macintosh; intel mac os x) excel/12.*
mozilla/5.0 (macintosh; u; *mac os x; *) applewebkit/* (*) pandora/2.*
mozilla/5.0 (snappreviewbot) gecko/* firefox/*
mozilla/5.0 (twiceler*)
mozilla/5.0 (windows; u; windows nt 5.1; en-us) speedy spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
mozilla/5.0 gurlchecker/*
mp3spider cn-search-devel at yahoo-inc dot com
mqbot*
msproxy/*
myzilla
naofavicon4ie*
net vampire/*
net_vampire*
netants*
netcarta_webmapper/*
netchart adv crawler*
netid.com bot*
netprospector*
netpumper*
netsucker*
netzip downloader*
newsgator/*
nextgensearchbot*(for information visit *)
nextools webagent*
ng-search/*
ng/*
nicebot
nozilla/p.n (just for ids woring)
np/*
npbot*
nso_debugger_user/2.0
nudelsalat/*
nutch/0.? (openx spider)
nutscrape
nutscrape/* (cp/m; 8-bit*)
nv32ts
obot
ocn-soc/*
offline downloader*
offline explorer*
online link validator (http://www.dead-links.com/)
open web analytics bot*
oracle enterprise search
ossproxy*
outfoxbot/*
p3p client
pagedown*
pageload*
pagenest/*
pajaczek/*
panscient.com
pavuk/*
pear http_request*
pete-spider/1.*
php*
picaloader*
pigblock (windows nt 5.1; u)*
pixfinder/*
plantynet_webrobot*
pmafind
pockey*
poe-component-client-http/*
polybot?*
privoxy/*
prowebwalker*
proxytester*
prozilla*
psbot/* (?http://www.picsearch.com/bot.html)
pycurl/*
python*
quickfinder crawler
radiation retriever*
realdownload/*
redcarpet/*
repomonkey*
rpt-httpclient/*
rssimagesbot/0.1 (*http://herbert.groot.jebbink.nl/?app=rssimages)
sbl-bot*
scollspider/2.*
scoutabout*
searchbot admin@google.com
seasydl/*
seeker.lookseek.com
seznambot/*
shaboyi spider
shareaza*
shelob (shelob@gmx.net)
shelob v1.*
sherlock/*
shim?crawler*
showxml/1.0 libwww/5.4.0
silentsurf*
site valet online*
siteparser/*
sitesnagger*
sitesucker/*
sitewinder*
slysearch/*
smallproxy*
smartdownload/*
sna-0.0.*
snapbot/*
snoopy*
softwing_tear_agent*
sogou develop spider/*
sogou head spider*
sogou js robot(*)
sogou orion spider/*
sogou pic agent
sogou pic spider/*
sogou push spider/*
sogou spider
sogou web spider*
sogou-test-spider/*
sohu*
space*bison/*
spankbot*
speeddownload/*
speedy spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
spider (tspyyp@tom.com)
sqeobot/0.*
squigglebotbot/*
sqworm/*
star*downloader/*
steeler/*
steroid download
strategic board bot (?http://www.strategicboard.com)
sunrise/0.*
superbot/*
superhttp/*
surf knight
surfcontrol
surveybot/*
synapticsearch/ai crawler 1.?
taiga web spider
talkro web-shot/*
tarantula/*
tasap-image-robot/0.* (http://www.tasap.com)
tcl http client package*
teleport*
terrawizbot/*
theinformant*
theme spider*
titanium 2005 (4.02.01)
tmcrawler
toata dragostea*
turnitinbot/*
tutorgigbot/*
tutorial crawler*
twingly recon
twisted pagegetter
twitturly*
uoftdb_experiment* (leehyun@cs.toronto.edu)
uri::fetch/*
url2file/*
user*agent:*
user_agent
usyd-nlp-spider*
utilmind httpget
vci webviewer*
vegas95/*
vengabot/*
virus_detector*
vobsub
wadaino.jp-crawler*
wap_browser/5.0 (compatible; yodaobot/1.*)
web downloader*
web downloader/*
web image collector*
web magnet*
webalta crawler/*
webauto/*
webbandit/*
webclipping.com
webcollage*
webcopier*
webcorp/*
webdownloader*
webenhancer*
webfetch
webfetch/*
webgatherer*
webget
webimages * (?http://herbert.groot.jebbink.nl/?app=webimages?)
webminer*
webpix*
webreaper*
webripper
websauger*
website downloader*
website extractor*
website quester
websiteextractor*
websnatcher*
webster pro*
webstripper*
webwhacker*
webzip*
west wind internet protocols*
wget*
winhttp*
winscripter inet tools
wintools
wire/* (linux*bot,robot,spider,crawler)
wisebot/*
wordpress-b-/2.*
wordpress-do-p-/2.*
woriobot*
www-mechanize/*
wwwster/* (beta, mailto:gue@cis.uni-muenchen.de)
xaldon webspider*
xenu* link sleuth*
xerka webbot v1.*
xspider*
y!oasis*
yahoo-mmcrawler*
yodaobot/*
yodaobot/1.* (*)
yoow!/* (?http://www.yoow.eu)
yrl_odp_crawler
zao-crawler
zao/*
zend_http_client
zibb crawler (email address / www address)

*

Other Records

Comments

• Provided courtesy of http://browsers.garykeith.com.
• Created on Wednesday, June 22, 2011 at 11:26 PM GMT.
• Place this file in the root public folder of your website.
• It will suggest to the following bots that they not index your website.
• WineWeb custom parameters

Warnings

• 28 invalid lines.