mikranet.org.il
robots.txt

Groups

mozilla/5.0 (compatible; seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)
addthis.com robot tech.support@clearspring.com
msrbot (http://research.microsoft.com/research/sv/msrbot/)
msrbot
sandcrawler
shim-crawler
scoutjet
mozilla/5.0 (compatible; ahrefsbot/5.0; +http://ahrefs.com/robot/)
mozilla/5.0 (compatible; dotbot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
dotbot/1.0.1 (http://www.dotnetdotcom.org/
mozilla/5.0 (compatible; dotbot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)
mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.searchtechnologies.com)
sputnikbot
accoona
addsugarspiderbot
anyapexbot
becomebot
beslistbot
billybobbot
bimbot
boitho
btbot
catchbot
cosmos
covario
cerberian
diamondbot
discobot
emeraldshield.com
envolk[its]spider
esperanzabot
exabot
fast enterprise crawler
fast-webcrawler
fdse robot
furlbot
fyberspider
g2crawler
gaisbot
galaxybot
geniebot
gigabot
girafabot
happyfunbot
holmes
htdig
iaskspider
ia_archiver
irlbot
issuecrawler
jaxified bot
jyxobot
koepabot
l.webis
lapozzbot
larbin
ldspider
ldspider
linguee bot
linkwalker
lmspider
lwp-trivial
mabontland
magpie-crawler
mj12bot
mogimogi
mojeekbot
moreoverbot
morning paper
mvaclient
mxbot
netresearchserver
netseer crawler
newsgator
ng-search
nicebot
nusearch
nutchcvs
nymesis
obot
oegp
omgilibot
omniexplorer_bot
oozbot
orbiter
pagebiteshyperbot
peew
polybot
postpost
psbot - image search
pycurl
qseero
radian6
rampybot
rufusbot
sbider
scrubby
searchsight
seekbot
semanticdiscovery
sensis web crawler
seochat::bot
shopwiki
shoula robot
silk
sitebot
snappy
speedy spider
sqworm
suggybot
surveybot
terrawizbot
thesubot
tineye/1.1 (http://tineye.com/crawler.html)
tineye
truwogps
turnitinbot
tweetedtimes bot
twengabot
updated
urlfilebot
vagabondo
vortex
voyager
vyu2
webcollage
websquash.com
wf84
womlpefactory
xaldon_webspider
yacy
yasaklibot
yooglifetchagent
zao/0.1 (http://www.kototoi.org/zao/)
mozilla/4.0 (compatible; zealbot 1.0)
zspider/0.9-dev http://feedback.redkolibri.com/
mozilla/4.0 compatible zyborg/1.0 dlc (wn.zyborg@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 dead link checker (wn.zyborg@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 dead link checker (wn.dlc@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn.zyborg@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn-16.zyborg@looksmart.net; http://www.wisenutbot.com)
mozilla/4.0 compatible zyborg/1.0 (wn-14.zyborg@looksmart.net; http://www.wisenutbot.com)

*

No rules defined. All paths allowed.

Other Records

Comments

• Block All trafic for certain user agents
• AhrefsBot
• Find Links
• TinEye
• Zyborg

Warnings

• 1 invalid line.