ausdauerleistung.de
robots.txt

Groups

*

Other Records

afilias web mining tool
ahrefsbot
aihitbot
aqua_products
asterias
b2w/0.1
backdoorbot/1.0
backlinkcrawler
baiduspider
blowfish/1.0
bookmark search tool
botalot
bpimagewalker
bpimagewalker*
bdbrandprotect
birubot
bixolabs
botonparade
bubing
builtbottough
bullseye/1.0
bullseye
bunnyslippers
catchbot
cheesebot
cherrypicker
cherrypickerse/1.0
cherrypickerelite/1.0
comodo ssl checker
comodo-certificates-spider
content crawler
copyrightcheck
cosmos
crescent internet toolpak http ole control v.1.0
crescent
dcpbot
discobot
dittospyder
ec2linkfinder
edisterbot
emailcollector
emailsiphon
emailwolf
erocrawler
eurobot
exabot
exdomain
extractorpro
ezooms
fairad client
findfiles.net
findlinks
foobot
gaisbot
getright/4.2
gigabot
gonzo
grub
grub-client
harvest/1.5
hloader
htdig
httplib
huaweisymantecspider
humanlinks
ia_archiver/1.6
ia_archiver
iccrawler - icjobs
ichiro
icjobs
infonavirobot
ips-agent
iron33/1.0.2
jennybot
jikespider
kaloogabot
kenjin spider
keyword density/0.9
larbin
lb-spider
lex
lexibot
libweb/clshttp
linkdex.com
linkextractorpro
linkscan/8.1a unix
linkscan
linkwalker
lnspiderguy
looksmart
lwp-trivial/1.34
lwp-trivial
magpie-crawler
mata hari
microsoft url control - 6.00.8169
microsoft url control - 5.01.4511
microsoft url control
miixpc/4.2
miixpc
mister pix
mj12bot
mlbot
moget/2.1
moget
msiecrawler
nerdbynature.bot
netants
netestate ne crawler
netmechanic
nicerspro
nutch
obot
offline explorer
oneriot
openbot
openfind data gathere
openfind
openindexspider
opidoobot
oracle ultra search
pagepeeker
perman
picmole
pixray-seeker
plukkie
propowerbot/2.14
prowebwalker
psbot
purebot
python-urllib
qualidator*
queryn metasearch
repomonkey bait & tackle/v1.01
repomonkey
reverseget
rma
schrein
scooter
scoutjet
screaming frog seo spider
search17
searchpreview
semrushbot
seznambot
seokicks-robot
seokicks
sistrix
sitebot
sitesnagger
slysearch
spankbot
spanner
spbot
speedy
spinn3r
suggybot
suzuran
swebot
szukacz/1.4
teleport
teleportpro
telesoft
the intraformantuser-agent: *thunderstone*
thenomad
tineye
true_robot/1.0
true_robot
tocrawl/urldispatcher
turingos
turnitinbot
unisterbot
unister*
unwindfetchor
updownerbot
url control
url_spider_pro
urly warning
vci webviewer vci webviewer win32
vci
voilabot
web image collector
webauto
webbandit
webcopier
webenhancer
webinator
webmastercoffee
webmasterworldforumbot
webreaper
webripper
websauger
wbsearchbot
website quester
webster pro
webstripper
webzip/4.0
webzip
weneobot
wget/1.6
wget/1.5.3
wget
www-collector-e
xenu's link sleuth 1.1c
xenu's
yacybot
yandex
yeti
yeti-mobile
zeus 32297 webster pro v2.9 win32
zeus link scout
zeus
zookabot
zyborg

Other Records

Comments

• unerwuenschte bots, die aber die robots.txt abfragen
• Despicable and evil robots to keep out :)
• monitor:
• "ssearch_bot (sSearch Crawler; http://www.semantissimo.de)"
• "Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)"
• "Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)"
• unerwünschte bots, die die robots.txt NICHT abfragen, gehören ggf. per Rewrite gesperrt:
• "Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)" IGNORIERT ROBOTS.TXT
• "Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)" IGNORIERT ROBOTS.TXT
• in der Regel okay:
• "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
• "Googlebot-Image/1.0"
• "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
• "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
• "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
• "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
• "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
• "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
• "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
• "Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)"
• "CloudACL/Nutch-1.4"
• "webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm)"
• "Mail.RU/2.0" (russ. Suchmaschine)
• "Sosospider+(+http://help.soso.com/webspider.htm)" (chin. Suchmaschine)
• "ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)" (hängt auch mit archive.org zusammen)
• "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
• "Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)"
• "Eurobot/1.1 (http://eurobot.ayell.eu)"
• "Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0" (koreanische Suchmaschine)
• "Acoon v4.10.3 (www.acoon.de)"
• "DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)" (jap. Suchmaschine)
• "ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)"
• "frogl-bot (Version: 1.06, powered by www.frogl.de +http://www.frogl.de/pfadzurbotseite/bot.html)"
• "Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)"
• "Agent-SharewarePlazaBot/3.0+(+http://www.SharewarePlaza.com)" IGNORIERT ROBOTS.TXT
• "Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)" IGNORIERT ROBOTS.TXT
• "www.freefileszone.com PadPollbot/1.1b (+http://www.freefileszone.com/)" IGNORIERT ROBOTS.TXT
• "Mozilla/5.0 (compatible; Sitedomain-Bot 1.0; Headers only; +http://www.sitedomain.de/sitedomain-bot/)" IGNORIERT ROBOTS.TXT - checkt auf gelöschte Domains - ruft nur Hauptseite auf
• "emefgebot/beta (+http://emefge.de/bot.html)" IGNORIERT ROBOTS.TXT

Warnings

• 1 invalid line.