dailyreadlist.com
robots.txt

Groups

adsarobot
ah-ha
ahrefsbot
aktuelles
almaden
amazonbot
amzn_assoc
anarchie
applebot
aspseek
assort
athens
atomz
attach
attache
autoemailspider
awariorssbot
awariosmartbot
backweb
baiduspider
baiduspider-image
baiduspider-video
bandit
barkrowler
batchftp
bdfetch
big.brother
blackwidow
blexbot
bmclient
boston project
bot mailto:craftbot@yahoo.com
bravobrian spiderengine marcopolo
buck
buddy
bullseye
bumblebee
bytespider
capture
cherrypicker
chinaclaw
cicc
clipping
coccocbot-image
coccocbot-web
collector
copier
crescent
crescent internet toolpak
custo
cyberalert
da
deweb
diagem
digger
digimarc
diibot
disco
disco pump
discofinder
dotbot
download demon
download wonder
downloader
drip
dsurf15a
dts.agent
easydl
ecatch
ecollector
efp@gmx.net
eirgrabber
email
email extractor
emailcollector
emailsiphon
emailwolf
express webpictures
extractorpro
eyenetie
fastlwspider
favorg
favorites sweeper
fetch
fezhead
filehound
flashget webwasher
flickbot
fluffy
frontpage
galaxybot
generic
getleft
getright
getsmart
getweb!
getwebpage
gigabaz
girafabot
go-ahead-got-it
go!zilla
gornker
gotit
grabber
grabnet
grafula
green research
grub-client
harvest
hhjhj@yahoo
hloader
hmview
homepagesearch
http generic
httpdown
httrack
httrack
ia_archiver
ibm_planetwide
image stripper
image sucker
imagefetch
imagesiftbot
incywincy
indy library
informant
ingelin
interget
internet ninja
internetlinkagent
internetseer.com
iria
irvine
jetcar
joc
joc web spider
justview
kwebget
lachesis
larbin
leechftp
lexibot
lftp
libwww
likse
link
links aromatized
linkwalker
lwp
lwp-trivial
mac finder
mag-net
magnet
mail.ru
mass downloader
mcspider
megaindex.ru
memo
microsoft.url
midown tool
mirror
missigua locator
mister pix
mj12bot
mmmtocrawl/urldispatcherlll
mojeekbot
ms frontpage
msfrontpage
msiecrawler
msproxy
multithreaddb
nationaldirectory
navroad
nearsite
net vampire
netants
netcarta
netmechanic
netprospector
netresearchserver
netspider
netzip
netzip downloader
netzippy
newt
nicerspro
ninja
npbot
octopus
offline explorer
offline navigator
omgili
opal
openfind
opentextsitecrawler
orangebot
packrat
pagegrabber
papa foto
paqlebot
pavuk
pcbrowser
personapilot
petalbot
ping
pingalink
pockey
proxy
psbot
psurf
puf
pump
pushsite
qrva
qwantify
realdownload
reaper
recorder
reget
replacer
repomonkey
robozilla
rover
rpt-httpclient
rsync
scooperbot
scooter
searchexpress
searchhippo
searchterms.it
second street research
seeker
semanticbot
semrushbot
serendeputybot
serpstatbot
seznambot
shai
siphon
sitecheck
sitecheck.internetseer.com
sitesnagger
slysearch
smartdownload
snagger
snake
spacebison
spegla
spiderbot
sproose
sqworm
stripper
sucker
superbot
superhttp
surfbot
surfwalker
szukacz
takeout
tarspider
teleport pro
templeton
trendictionbot
truerobot
tv33_mercator
uiowacrawler
url_spider_pro
urlspiderpro
utilmind
vacuum
vagabondo
vayala
visibilitygap
voideye
vspider
w3mir
web data extractor
web downloader
web image collector
web sucker
web.by.mail
webauto
webbandit
webclipping
webcollage
webcollector
webcopier
webcraft@bea
webdevil
webdownloader
webdup
webemailextrac
webfetch
webgo is
webhook
webinator
webleacher
webmasters
webminer
webmirror
webmole
webreaper
websauger
website
website extractor
website quester
websnake
webster
webstripper
websucker
webvac
webwalk
webweasel
webwhacker
webzip
wget
whacker
whizbang
whostalking
widow
wisebot
wumpus
wweb
wwwoffle
x-tractor
xaldon webspider
xenu
xget
yandex
yandexbot
youdaobot
zeus

*

Other Records