p2r-expert.com
robots.txt

Groups

mozilla/5.0 (compatible; seznam screenshot-generator 2.1;+http://fulltext.sblog.cz/screenshot/)
addthis.com robot tech.support@clearspring.com
msrbot (http://research.microsoft.com/research/sv/msrbot/)
msrbot
sandcrawler
shim-crawler
scoutjet
ahrefsbot
mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)
mozilla/5.0 (compatible; ahrefsbot/6.1; +http://ahrefs.com/robot/)
mozilla/5.0 (compatible; dotbot/1.1; http://www.dotnetdotcom.org/,crawler@dotnetdotcom.org)
dotbot/1.0.1 (http://www.dotnetdotcom.org/
mozilla/5.0 (compatible; dotbot/1.1;http://www.opensiteexplorer.org/dotbot, help@moz.com)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
mozilla/4.0 compatible zyborg/1.0 dlc (wn.zyborg@looksmart.net;http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 dead link checker (wn.zyborg@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 dead link checker (wn.dlc@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn.zyborg@looksmart.net;http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn-16.zyborg@looksmart.net;http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn-14.zyborg@looksmart.net;http://www.wisenutbot.com)
accoona
addsugarspiderbot
alexibot
alphaseobot
alphaseobot-sa
alvinetspider
antenne hatena
anyapexbot
apocalxexplorerbot
asterias
backdoorbot/1.0
baidu
baiduspider
baiduspider-ads
baiduspider-cpro
baiduspider-favo
baiduspider-image
baiduspider-news
baiduspider-video
becomebot
beslistbot
billybobbot
bimbot
bizinformation
black hole
blexbot
blowfish/1.0
boitho
botalot
btbot
builtbottough
bullseye/1.0
bunnyslippers
catchbot
cegbfeieh
cerberian
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
copyrightcheck
cosmos
covario
crescent
crescent internet toolpak http ole control v.1.0
diamondbot
disco pump 3.1
discobot
dittospyder
dotbot
emailcollector
emailsiphon
emailwolf
emeraldshield.com
envolk[its]spider
erocrawler
esperanzabot
exabot
extractorpro
fast enterprise crawler
fast-webcrawler
fdse robot
flamingo_searchengine
foobot
furlbot
fyberspider
g2crawler
gaisbot
galaxybot
geniebot
gigabot
girafabot
grapeshot
happyfunbot
harvest/1.5
hloader
holmes
htdig
httplib
httrack
httrack 3.0
humanlinks
ia_archiver
iaskspider
igentia
infonavirobot
irlbot
issuecrawler
jaxified bot
jennybot
jikespider
jyxobot
kenjin spider
koepabot
l.webis
lapozzbot
larbin
ldspider
lexibot
libweb/clshttp
linguee bot
linkextractorpro
linkscan/8.1a unix
linkwalker
lmspider
lwp-trivial
lwp-trivial/1.34
mabontland
magpie-crawler
mata hari
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
miixpc
miixpc/4.2
mister pix
mj12bot
mlbot
moget
moget/2.1
mogimogi
mojeekbot
moreoverbot
morning paper
mozilla/4.0 (compatible; zealbot 1.0)
mozilla/5.0 (compatible; heritrix/1.14.3+http://www.searchtechnologies.com)
ms search 4.0 robot
ms search 5.0 robot
mvaclient
mxbot
naverbot
netants
netattache
netmechanic
netresearchserver
netseer crawler
newsgator
ng-search
nicebot
nicerspro
nusearch
nutchcvs
nymesis
obot
oegp
offline explorer
omgilibot
omniexplorer_bot
oozbot
openfind
openindexspider
orbiter
pagebiteshyperbot
peew
polybot
postpost
propowerbot/2.14
prowebwalker
psbot
psbot - image search
pycurl
qseero
quepasacreep
queryn metasearch
radian6
rampybot
repomonkey
rma
rufusbot
sbider
scrubby
searchsight
seekbot
semanticdiscovery
sensis web crawler
seochat::bot
shopwiki
shoula robot
sightupbot
silk
sitebot
sitebot
sitesnagger
sitesucker
snappy
sogou web spider
sosospider
spankbot
spanner
speedy
speedy spider
sputnikbot
sqworm
suggybot
superbot
superbot/2.6
surveybot
suzuran
szukacz/1.4
teleport
telesoft
terrawizbot
the intraformant
thenomad
thesubot
tighttwatbot
tineye
tineye/1.1 (http://tineye.com/crawler.html)
titan
tocrawl/urldispatcher
toscrawler
trendictionbot
true_robot
true_robot/1.0
truwogps
turingos
turnitinbot
tweetedtimes bot
twengabot
updated
urlfilebot
urlpouls
urly warning
vagabondo
vci
vortex
voyager
vyu2
web image collector
webauto
webbandit
webbandit/3.50
webcollage
webcopier
webcopy
webenhancer
webmasterworldforumbot
webmirror
webreaper
websauger
website extractor
website quester
websquash.com
webster pro
webstripper
webstripper/2.02
webzip
wf84
wget
wikiofeedbot
winhttrack
womlpefactory
www-collector-e
xaldon_webspider
xenu link sleuth/1.3.8
yacy
yandex
yasaklibot
yooglifetchagent
yrspider
zao/0.1 (http://www.kototoi.org/zao/)
zeus
zookabot
zspider/0.9-dev http://feedback.redkolibri.com/

*

Other Records

Comments

• Block All trafic for certain user agents
• AhrefsBot
• Link tracker
• Find Links
• Zyborg

Warnings

• 1 invalid line.