diomi.es
robots.txt

Groups

*

adbeat_bot
addthis
adsbot
ahrefsbot
ahrefsbot/1.0
ahrefsbot/2.0
ahrefsbot/3.0
ahrefsbot/3.1
ahrefsbot/4.0
ahrefsbot/5.0
ahrefsbot/5.1
ahrefsbot/5.2
ahrefsbot/7.0
ahrefsbot/7.1
ahrefsbot/7.2
ahrefsbot/8.0
ahrefsbot/9.0
ahrefscrawler
ahrefssiteaudit
aihitbot
amazonbot
amznkassocbot/4.0
anews
anews/1.0
aport
applebot
aspiegelbot
awariosmartbot
baidu
baiduspider
baiduspider-image
barkrowler
bdcbot
bdcbot/1.0
bitlybot
bitlybot/3.0
blexbot
blexbot/1.0
boitho
bsalsa
bubing
buck
butterfly
ccbot
ccbot/2.0
censysinspect
changedetection
cincraw
coccoc
coccocbot
crawler4j
crazywebcrawler-spider
criteobot/0.1
daum
daumoa
detectify
deusu/5.0.2
disco pump
docomo/2.0
dotbot
dotbot
dotbot/1.1
easouspider
emailcollector
emailsiphon
emailwolf
exabot
exabot/3.0
ezinearticleslinkscanner
ezooms
findxbot
findxbot/1.0
flipboard
flipboardproxy
geedobot
genieo
germcrawler
getintent crawler
gigabot
gingercrawler
gluten free crawler
go-http-client
grapeshot
greenbrowser
grouphigh
grouphigh/1.0
guzzlehttp
guzzlehttp/6.1.0
haosouspider
heritrix
honolulubot
hypercrawl
ia_archiver
ichiro
ichiro/3.0
ichiro/4.0
infoweb
iskanie
istellabot
kocmohabt
kraken
kraken/0.1
kribrum
kribrum/0.1
libwww-perl
linguee bot
linkdexbot
linkdexbot/2.0
linkdexbot/2.1
linkdexbot/2.2
linkedinbot
linkpadbot
ltx71
ltx71 - (http://ltx71.com/)
mail.ru_bot
masscan
mauibot
maxpointcrawler
meanpathbot
mediatoolkitbot
megaindex
megaindex
megaindex.ru
megaindex.ru
megaindex.ru/2.0
mixnodecache
mj12bot
mj12bot/v1.0.5
mj12bot/v1.0.6
mj12bot/v1.0.7
mj12bot/v1.0.8
mj12bot/v1.2.3
mj12bot/v1.2.4
mj12bot/v1.4.0
mj12bot/v1.4.3
mj12bot/v1.4.5
mj12bot/v1.4.6
mj12bot/v1.4.7
mj12bot/v1.4.8
mojeekbot
monsidobot
naverbot
nerdybot
netinfobot
netseer
netsystemsresearch
newspaper
nimbostratus-bot
nutch
nutch bots
obot
offline explorer
orbbot
pagething
paperlibot
paperlibot/2.1
petalbot
phpservermon/3.1.1
pinterestbot
privacyawarebot/1.1
proximic
psbot
queryseekerspider
qwantify
riddler
rogerbot
rogerbot/1.0
rsiteauditor
rytebot
safednsbot
scooperbot
scooter
scoutjet
scrapy
scrapy/1.2.2
screaming frog seo spider
screenerbot
searchmetricsbot
seekport crawler
seekportbot
semrushbot
semrushbot-ba
semrushbot-bm
semrushbot-ct
semrushbot-sa
semrushbot-sa/0.97
semrushbot-si
semrushbot-swa
semrushbot/1.1
semrushbot/1.1~bl
sentibot
seokicks
serpstatbot
seznambot
seznambot/3.2
sf/2 (skipfish)
siteauditbot
sitecheckerbotcrawler
siteexplorer
sogou
sogou head spider/3.0
sogou orion spider/3.0
sogou pic agent
sogou pic spider/3.0
sogou web spider
spbot
spbot/5.0.3
special_archiver
special_archiver/3.1.1
splitsignalbot
sputnikbot
sputnikbot/2.3
startmebot
statusoid
steeler
steeler/3.5
stormcrawler
sunrise
surdotlybot
swiftbot
tdjbot
teleport
teleport pro
testomatobot
the knowledge ai
tpradstxtcrawler
trendictionbot
trendictionbot0.5.0
trident
trident/5.0
trident/6.0
trident/7.0
trovitbot
turnitinbot
tweetmeme
tweetmemebot
twengabot
twengabot-2.0
twengabot-discover
twengabot/2.0
twiceler
um-fc
um-fc/1.0
um-ic
um-ic/1.0
um-ln
um-ln/1.0
unwindfetchor
uptimebot
uptimebot/1.0
uptimerobot
uptimerobot/2.0
urlbot
voltron
wbsearchbot
web-by-mail
webartexbot
webdatastats
websnake
webzip
wellknownbot
wesee
wget
wget
wikido
woorankreview
wotbox
x11
xovibot
y! j-asr
yacybot
yadirectbot
yadirectfetcher
yak
yandex
yandexbot
yandexdirect
yandexmarket
yellowbrandprotectionbot
yeti
yeti/1.1
yodaobot
yoozbot-2.2
youdaobot/1.0
zgrab
zoombot
zoominfobot

Other Records

Warnings

• 6 invalid lines.