twelvebeauty.com
robots.txt

Groups

mj12bot

mediapartners-google*

israbot

orthogaffe

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

fast

wget

grub-client

k2spider

npbot

webreaper

accoona
adbeat_bot
aghaven
ahrefsbot
ahrefssiteaudit
aipbot
aipbot*
aipbot/1.0
alexa
alexa bitlybot
alexibot
altavista intranet v2.0 avs eval search@freeit.com
altavista intranet v2.0 compaq altavista eval sveand@altavista.net
altavista intranet v2.0 evreka.com crawler@evreka.com
altavista v2.0b crawler@evreka.com
alvinetspider
amfibibot
anonymous
antenne hatena
antibot
apocalxexplorerbot
appengine
appie
aqua_products
archive
argus/1.1
artabus
asterias
atspider
attentio
av fetch 1.0
avsearch-3.0(altavista/avc)
aws cloud based
b2w
b2w/0.1
backdoorbot
backdoorbot/1.0
backlinkcrawler
baiduspider
becomebot
becomebot
bigbrother
biglotron
biglotron (beta 2;gnu/linux)
bizinformation
black hole
black.hole
blackwidow
blekkobot
blexbot
blexbot
blowfish
blowfish/1.0
boardpulse
boitho.com-dc
bookmark search tool
bot mailto:craftbot@yahoo.com
bot/1.0
botalot
botrighthere
brandprotect
bruinbot
builtbottough
bullseye
bullseye/1.0
bunnyslippers
butterfly
catchbot
cazoodlebot
ccbot
ccubee
ccubee/3.5
cegbfeieh
cfetch
cfetch/1.0
chatgpt
chatgpt-user
cheesebot
cherrypicker
cherrypicker /1.0
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
chroot
collage
combine
cometrics-bot
complex_network_group
convera
convera internet spider v6.x
converacrawler
converacrawler/0.2
converacrawler/0.9d
converamultimediacrawler
converamultimediacrawler/0.1
coolbot
copernic
copyrightcheck
cosmos
crescent
crescent internet toolpak http ole control v.1.0
crescent internet toolpak httpole control v.1.0
curl
custo
cydralspider
deepnet explorer
default.ida
digext
dimensionet
disco
disco pump
disco pump 3.0
disco pump 3.1
disco pump 3.2
discobot
discofinder
dittospyder
docomo
dotbot
dotbot
dotbot/1.1
download demon
download demon/3.2.0.8
download demon/3.5.0.11
download wonder
doyoucheckbot
drecombot
dsurf
dtaagent
dulance bot
dumbot
e-societyrobot
ecatch
ecatch/3.0
echo!
echo!/2.0
eirgrabber
elitesys entry
email extractor
emailcollector
emailsiphon
emailsmartz
emailwolf
enterprise_search
enterprise_search/1.0
envolk
erocrawler
erowz
es
esirover
everbeecrawler
exabot
exabot
exabot-images
exabot/2.0
express webpictures
express webpictures (www.express-soft.com)
extractorpro
eyenetie
ezooms
fairad client
fairshare
fasterfox
fdse
findlinks
flaming attackbot
flamingo_searchengine
flashget
flashget webwasher 3.2
foobot
freefind
freewebmonitoring sitechecker/0.1
frontpage
frontpage [nc,or]
furlbot
g2crawler
g2reader-bot/1.0
gaisbot
gaisbot/3.0
geniebot
getbot
getright
getright/2.11
getright/3.1
getright/3.2
getright/3.3
getright/3.3.3
getright/3.3.4
getright/4.0.0
getright/4.1.0
getright/4.1.1
getright/4.1.2
getright/4.2
getright/4.2b (portuguxeas)
getright/4.2c
getright/4.3
getright/4.5
getright/4.5a
getright/4.5b
getright/4.5b1
getright/4.5b2
getright/4.5b3
getright/4.5b6
getright/4.5b7
getright/4.5c
getright/4.5d
getright/4.5e
getright/5.0beta1
getright/5.0beta2
geturl
getweb!
gigabot
gigabot
gigabot/3.0
go-ahead-got-it
go-http-client
go!zilla
go!zilla (www.gozilla.com)
go!zilla 3.3 (www.gozilla.com)
go!zilla 3.5 (www.gozilla.com)
grabnet
grafula
grub
gsa-crawler
hackertarget.com
harvest
harvest/1.5
hatena antenna
havindex
heritrix
hloader
hmview
hoowwwer
http://www.searchengineworld.com bot
http://www.webmasterworld.com bot
httpful/0.2.20
httplib
httrack
httrack [nc,or]
httrack 3.0
httrack 3.0x
humanlinks
ia_archiver
ia_archiver/1.6
ichiro
iconsurf
igentia
image collector
image stripper
image sucker
indy library
infonavirobot
infospiders
innosense
interget
internet explore
internet ninja
internet ninja 4.0
internet ninja 5.0
internet ninja 6.0
internetsupervision
ipselonbot
irlbot
iron
iron33/1.0.2
jakarta commons
jamesbot
java
jeeves
jennybot
jetbot
jetbot/1.0
jetcar
jikespider
jobcrawlerbot
jobo
jobrapido
joc web spider
jorgee
jyxobot
k2spider
kalooga
kavamringcrawler
kdd exploror
kenjin spider
kenjin.spider
keyword density
keyword density/0.9
keyword.density
larbin
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
larbin_2.6.2 (kabura@sushi.com)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (listonatccdotgatechdotedu)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 kabura@sushi.com
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 larbin2.6.2@unspecified.mail
larbin_2.6.2 listonatccdotgatechdotedu
larbin_2.6.2 vitalbox1@hotmail.com
lbot
leechftp
lexibot
libweb/clshttp
libwww-perl
lightningdownload
linguee
linkedin
linkextractorpro
linknzbot
linknzbot 2004
linknzbot*
linkpadbot
linkscan
linkscan/8.1a unix
linkscan/8.1a.unix
linksmanager
linksmanager
linksmanager.com_bot
linkwalker
ljseek
lmspider
lnspiderguy
looksmart
lwp
lwp-trivial
lwp-trivial/1.34
lwp*
magpie-crawler
mail sweeper
marketwirebot
mass downloader
mass downloader/2.2
mata hari
mata.hari
megaindex.ru
megaindex.ru/2.0
megalodon
metagerbot
metauri
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
microsoft url control - 6.01.9782
microsoft url control ãƒâ¢ã¢â€šâ¬ã¢â‚¬å“ 5.01.4511
microsoft url control ãƒâ¢ã¢â€šâ¬ã¢â‚¬å“ 6.00.8169
microsoft url control*
microsoft.url
midown tool
miixpc
miixpc/4.2
minibot(naverrobot)/1.0
missigua locator
mister pix
mister pix ii 2.01
mister pix ii 2.02a
mister pix version.dll
mister.pix
mlbot
moget
moget/2.1
mozilla
mozilla
mozilla/2.0 (compatible; ask jeeves)
mozilla/3
mozilla/4
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 2000)
mozilla/4.0 (compatible; msie 4.0; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 98)
mozilla/4.0 (compatible; msie 4.0; windows me)
mozilla/4.0 (compatible; msie 4.0; windows nt)
mozilla/4.0 (compatible; msie 4.0; windows xp)
mozilla/5
mozilla/5.0 (compatible; jobrapido/1.1; +http://www.jobrapido.com)
mozilla/5.0 (jobrapido webpump)
mrsputnik
ms search 4.0 robot
ms search 5.0 robot
msrbot
munky
myfamilybot
naver
naverbot
naverbot
naverbot-1.0
navroad
nearsite
nerdybot
net vampire
net vampire/3.0
netants
netants/1.10
netants/1.23
netants/1.24
netants/1.25
netattache
netattache light 1.1
netcraft web server survey
netmechanic
netresearchserver
netspider
netzip
netzip downloader 1.0 win32(nov 12 1998)
netzip-downloader
netzip-downloader/1.0.62 (win32; dec 7 1998)
netzippy+(http://www.innerprise.net/usp-spider.asp)
netzippy+(http:/www.innerprise.net/usp-spider.asp)
nextgensearchbot
nicerspro
nimblecrawler
ning/1.0
noxtrumbot
npbot
npbot/3
nutch
nutch*
nutchcvs
nutchcvs/0.06-dev
nutchcvs/0.7.1
nutchorg
nutraspace
obot
ocelli
octopus
offbyone
offline explorer/1.2
offline explorer/1.4
offline explorer/1.6
offline explorer/1.7
offline explorer/1.9
offline explorer/2.0
offline explorer/2.1
offline explorer/2.3
offline explorer/2.4
offline explorer/2.5
offline navigator
offline.explorer
omniexplorer_bot
oneriot
openai
openbot
openfind
openfind data gathere
openfind data gatherer
openindexspider
openintelligencedata
oracle ultra search
outfoxbot/0.5
owlin bot
pagegrabber
papa foto
pavuk
pbwf
pcbrowser
penthesilea
perman
petalbot
pgbot
phpdig
pingdom gigrib (http://www.pingdom.com)
pingdom.com_bot
pompos
postrank
powermarks
propowerbot
propowerbot/2.14
prowebwalker
psbot
psycheclone
psycheclone
python-requests
python-urllib
quepasacreep
quepasacreep
queryn metasearch
queryn.metasearch
r6_commentreader
r6_feedfetcher
radian6
radian6 comment reader
radian6 feedfetcher
radiation retriever
radiation retriever 1.1
rb2b-bot
realdownload
realdownload/4.0.0.40
realdownload/4.0.0.41
realdownload/4.0.0.42
reget
repomonkey
repomonkey bait & tackle/v1.01
repomonkey bait & tackle
repomonkey bait & tackle/v1.01
repomonkey bait & tackle/v1.01
research-spider
rma
robozilla
rogerbot
roverbot
rufusbot
sbider
schibstedsokbot
scooter
scooter_bh0-3.0.3
scooter_trk3-3.0.3
scooter-3.0.eu
scooter-3.0.fs
scooter-3.0.hd
scooter-3.0.vns
scooter-3.0qi
scooter-3.2
scooter-3.2.bt
scooter-3.2.dil
scooter-3.2.ex
scooter-3.2.jt
scooter-3.2.niv
scooter-3.2.sf0
scooter-3.2.snippet
scooter-3.3dev
scooter-ars-1.1
scooter-ars-1.1-ih
scooter-venus-3.0.vns
scooter-w3-1.0
scooter-w3.1.2
scooter/1.0
scooter/1.0 scooter@pa.dec.com
scooter/1.1 (custom)
scooter/2.0 g.r.a.b. v1.1.0
scooter/2.0 g.r.a.b. x2.0
scooter/3.3
scooter/3.3_sf
scooter/3.3.qa.pczukor
scooter/3.3.vscooter
scooter2_mercator_x-x.0
scoutjet
screaming frog seo spider
scrubby
scspider
searchdaimon.com-dc
searchmetricsbot
searchpreview
seekbot
seekbot
seekbot/1.0
semalt.com
semanticdiscovery
semrushbot
semrushbot
semrushbot-sa
seokicks-robot
seoprofiler
seznambot
shai&
shim-crawler
shopwiki
shopwiki/1.0
sightupbot
silk
sistrix
sistrix crawler
sitebot
sitesucker
slurp
slurp china
slysearch
smartdownload
smartdownload/1.2.76 (win32; apr 1 1999)
smartdownload/1.2.77 (win32; aug 17 1999)
smartdownload/1.2.77 (win32; feb 1 2000)
smartdownload/1.2.77 (win32; jun 19 2001)
snapbot
snappy
socialshare/1.0
softlayer server
sogou web spider
sootle
sosospider
spankbot
spanner
spbot
speedy
speedy spider
spiderbot
spiderbot/nutch-1.7
sproose
sqworm
sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux
ssearcher100
stanford
stanford comp sci
stanford compclub
stanford compsciclub
stanford spiderboys
steeler
suggybot
superbot
superbot/2.6
superbot/3.0 (win32)
superbot/3.1 (win32)
superhttp
superhttp/1.0
surfbot
surveybot
surveybot_ignoreip
suzuran
szukacz
szukacz/1.4
takeout
tarantula
teleport pro
teleport pro/1.29
teleport pro/1.29.1590
teleport pro/1.29.1634
teleport pro/1.29.1718
teleport pro/1.29.1820
teleport pro/1.29.1847
telesoft
templeton
teoma
the intraformant
the.intraformant
thenomad
theophrastus
tighttwatbot
titan
tocrawl
tocrawl/urldispatcher
toscrawler
trendictionbot
tridentspider
trovitbot
true_robot
true_robot/1.0
turingos
turnitinbot
turnitinbot/1.5
tweetmeme
twengabot
twiceler
twiceler
typhoeus
ultraseek
url control
url_spider_pro
urldispatcher
urlpouls
urly warning
urly.warning
vagabondo
vci
vci webviewer vci webviewer win32
verticrawlbot
vobsub
voideye
voilabot
voyager
voyager/1.0
vscooter
w3mir
watchdog/3.0
web image collector
web reaper
web sucker
web.image.collector
webauto
webauto/3.40 (win98; i)
webbandit
webbandit/3.50
webcapture
webcapture 2.0
webcatcher
webcopier
webcopier v.2.2
webcopier v2.5
webcopier v2.6
webcopier v2.7a
webcopier v2.8
webcopier v3.0
webcopier v3.0.1
webcopier v3.2
webcopier v3.2a
webcopy
webcopy
webcrawl.net
webemailextrac
webemailextrac.*
webenhancer
webfetch
webfetch/2.1.0
webfetcher
webgo is
webindexer
webleacher
webmasterworld extractor
webmasterworldforumbot
webmirror
webmirror
webreaper [info@webreaper.net]
webreaper [webreaper@otway.com]
webreaper v9.1 - www.otway.com/webreaper
webreaper v9.7 - www.webreaper.net
webreaper v9.8 - www.webreaper.net
webreaper vwebreaper v7.3 - www,otway.com/webreaper
websauger
websauger 1.20b
websauger 1.20j
websauger 1.20k
website extractor
website extractor (http:/www.asona.org)
website quester
website quester - www.asona.org
website quester - www.esalesbiz.com/extra/
website.quester
webster pro
webster.pro
webstripper/2.02
webstripper/2.03
webstripper/2.10
webstripper/2.12
webstripper/2.13
webstripper/2.15
webstripper/2.16
webstripper/2.19
webvac
webvac
webvulncrawl
webvulnscan
webwalk
webwasher
webwhacker
webzip
webzip/2.75 (http:/www.spidersoft.com)
webzip/3.65 (http://www.spidersoft.com)
webzip/3.80 (http://www.spidersoft.com)
webzip/4.0
webzip/4.0 (http://www.spidersoft.com)
webzip/4.1 (http:/www.spidersoft.com)
webzip/4.21
webzip/4.21 (http:/www.spidersoft.com)
webzip/5.0
webzip/5.0 (http:/www.spidersoft.com)
webzip/5.0 pr1 (http://www.spidersoft.com)
wget
wget
wget/1.10.2
wget/1.5.2
wget/1.5.3
wget/1.6
wget/1.8
wget/1.8.1
wget/1.8.2
wget/1.9-beta
whitevector crawler
whitevector+crawler
widow
wijubot
wijubot/1.0
wikiofeedbot
wikiwix-bot-3.0
willow
winhttrack
wise-guys
wise-guys
woozweb-monitoring
woriobot
www-collector-e
www-mechanize
wwwoffle
xaldon webspider
xaldon webspider 2.5.b3
xenu link sleuth
xenu link sleuth/1.3.8
xenu's
xenu's link sleuth 1.1c
xenu&
xenu&
xget
xirq
yacy
yandex
yandexbot
yandexcatalog
yandexdirect
yandexfavicons
yandeximages
yeti
yodaobot
youdaobot
yrspider
zebot
zebot_www.ze.bz
zeus
zeus 11389 webster pro v2.9 win32
zeus 11652 webster pro v2.9 win32
zeus 18018 webster pro v2.9 win32
zeus 26378 webster pro v2.9 win32
zeus 30747 webster pro v2.9 win32
zeus 32297 webster pro v2.9 win32
zeus 39206 webster pro v2.9 win32
zeus 41641 webster pro v2.9 win32
zeus 44238 webster pro v2.9 win32
zeus 51070 webster pro v2.9 win32
zeus 51674 webster pro v2.9 win32
zeus 51837 webster pro v2.9 win32
zeus 63567 webster pro v2.9 win32
zeus 6694 webster pro v2.9 win32
zeus 71129 webster pro v2.9 win32
zeus 82016 webster pro v2.9 win32
zeus 82900 webster pro v2.9 win32
zeus 84842 webster pro v2.9 win32
zeus 90872 webster pro v2.9 win32
zeus 94934 webster pro v2.9 win32
zeus 95245 webster pro v2.9 win32
zeus 95351 webster pro v2.9 win32
zeus 97371 webster pro v2.9 win32
zeus link scout
zookabot
zyborg

*

noxtrumbot

No rules defined. All paths allowed.

Other Records

msnbot

No rules defined. All paths allowed.

Other Records

slurp

No rules defined. All paths allowed.

Other Records

twitterbot

googlebot-image

adsbot-google

googlebot-mobile

Other Records

Comments

• Fichero robots.txt de Art Project Group (https://artprojectgroup.es) para WordPress.
• Si lo usas no olvides indicar su procedencia ;-)
• Última actualización: 15/12/2023.
• Código procedente de Wikipedia (https://en.wikipedia.org/robots.txt):
• Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
• and ignoring 429 ratelimit responses, claims to respect robots:
• http://mj12bot.com/
• advertising-related bots:
• Wikipedia work bots:
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Misbehaving: requests much too fast:
• Sorry, wget in its recursive mode is a frequent problem.
• Please read the man page and use it properly; there is a
• --wait option you can use to set the delay between hits,
• for instance.
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• Hits many times per second, not acceptable
• http://www.nameprotect.com/botinfo.html
• A capture bot, downloads gazillions of pages with no public benefit
• http://www.webreaper.net/
• Es hora de bloquear a otro buen puñado de bad bots y bots que generan consultas abusivas (818 User-agent diferentes).
• Algunas fuentes:
• https://sustainablewww.org/principles/block-unwanted-and-spammy-bots-with-robotstxt-and-speed-up-your-website
• https://www.cocooncenter.es/robots.txt
• https://www.keyadvice.co.uk/robots.txt
• http://rfec.com/robots.txt
• Si has llegado hasta aquí es que eres un buen bot así que toma y lee:
• Pero no todo:
• Ninguna carpeta que empiece por wp-:
• Nada de esto no te interesa:
• Nada de contenido dinámico:
• Nada de búsquedas:
• Ningún adjunto:
• Nada de feed:
• Ni carpetas que no te interesan:
• Limitamos las consultas excesivas de Yahoo!, Noxtrum y el bot de MSN:
• Estos bots deben de entrar hasta la cocina:
• Y aqui­ tienes nuestros sitemaps:
• ¡Y eso es todo amigo!

Warnings

• 1 invalid line.