mueblesleandro.com
robots.txt

Groups

acrobat
aisearchbot
baidu*
baiduspider
baiduspider+(+http://www.baidu.com/search/spider.htm)
baiduspider+(+http://www.baidu.jp/spider/)
baiduspider-image
baiduspider-video
black hole
blackwidow
blackwidow 4.40
cfetch/1.0
cheesebot
cherrypicker
comodo+http(s)+crawler
converamultimediacrawler/0.1
dloader(naverrobot)
dloader(speedy spider)
dotbot
emailcollector
emailsiphon
emailwolf
everbeecrawler
extractorpro
flash processor
flash+processor
flatlandbot
flatlandbot/baypup
grub-client
hloader
home.thenewweb.com
htdig/3.1.5
httrack
ichiro
ichiro
icollect
igetter
imagewalker
industry program
indy
indy library
innerprise
installshield
internetlinkagent/
intscanner
ipd
ipiumbot
iria
iupui research bot
java
java1
java1.3.0
java1.3.1
java2
jobo
joc web spider
johnhasbeenhere
jscript+processor
kapere
lachesis
larbin
larbin_2.6.1
larbin_2.6.1 larbin2.6.2@unspecified.mail
larbin_2.6.2
larbin_2.6.2 larbin2.6.2@unspecified.mail
leechget
libwww-perl
lightningdownload
linkalarm
linkchecker
linklint-checkonly
linkman
llupdatectrl
mac finder
mail sweeper
mass
mass downloader
mcbot
metaproducts
metaproducts download express
mfc_tear_sample
mfhttpscan
moget
moget
mozilla/4.0 (compatible; naverbot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
mozilla/5.0 (compatible; yandexblogs/0.99; robot; b; +http://yandex.com/bots)
mozilla/5.0 (compatible; yandexbot/3.0; +http://yandex.com/bots)
mozilla/5.0 (compatible; yandexbot/3.0; mirrordetector; +http://yandex.com/bots)
mozilla/5.0 (compatible; yandexmedia/3.0; +http://yandex.com/bots)
mygetright
naver
naverbot
naverrobot
netpumper
newt
nextgensearchbot
nicerspro
nitro
nitro downloader
nudelsalat
nutch
obot
offline
offline explorer
page_prefetcher
pagmiedownload
pavuk
pixgrabber
plantynet_webrobot
plucker
pockey
popdexter
program
program shareware
progressive
progressive download
prowebwalker
proxytester
psbot
puf
puxarapido
python-urllib
python-webchecker
realdownload
repomonkey
repomonkey bait & tackle
robotmidareru
rpt-httpclient
scat
scoutabout
semanticdiscovery
siphon
sitesnagger
sitesnagger
sitewinder
slurp
slysearch
smartdownload
softwing_tear_agent
sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou pic agent
sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou spider
sogou spider
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
sogou-test-spider/4.0 (compatible; msie 5.5; windows 98)
sonic
sosospider
speeddownload
speedy
sprocket
sq
sq webscanner
stamina
star
star downloader
steeler
superhttp
surveybot
synobot
teleport
teleport
teleport pro
thunderstone
turnitinbot
turnitinbot
tweakmaster
twiceler
udmsearch
undisclosed
urlgetfile
utilmind
utilmind httpget
vcikjzddls
vobsub
voyager/2.0
voyager-hc/1.0
web downloader
webalta
webauto
webcapture
webclipping.com
webcollage
webcopier
webcopier
webinator
webleacher
webmole
webreaper
websauger
website extractor
webster
webstripper
webstripper
webzip
webzip
wep search
wep search 00
wget
whizbang
whsearch
wildsoft
wildsoft surfer
winhttp.winhttprequest
woriobot
www4mail
wwwoffle
xaldon
xaldon webspider
xedit
xenu
yandex
yandex*
yandexsomething/1.0
yanga worldsearch bot v1.1/beta
yanga worldsearch bot v1.1/beta (http
yeti
youdaobot
zao
zbot
zeus
zyborg

*

Comments

• robots.txt automaticaly generated by PrestaShop e-commerce open-source solution
• http://www.prestashop.com - http://www.prestashop.com/forums
• This file is to prevent the crawling and indexing of certain parts
• of your site by web crawlers and spiders run by sites like Yahoo!
• and Google. By telling these "robots" where not to go on your site,
• you save bandwidth and server resources.
• For more information about the robots.txt standard, see:
• http://www.robotstxt.org/wc/robots.html
• User-agent: ia_archiver
• ---------------------------------------
• Files

Warnings

• 5 invalid lines.