onlyfindersearch.com
robots.txt

Groups

*

haosouspider

baiduspider

mail.ru_bot

mj12bot

qwantify

sogou web spider

adbeat_bot

admantx

ahrefsbot

alexabot
ia_archiver
archive.org_bot
ia_archiver-web.archive.org

awariosmartbot
awariorssbot

blexbot

builtwith

domaincrawler

dotbot

grapeshotcrawler
grapeshot

megaindex.ru

pinterest

proximic

semrushbot
semrushbot-sa

serpstatbot

seznambot

siteexplorer

spbot

velenpublicwebcrawler

sistrix
sistrix crawler

xovibot

yisouspider

nimbostratus-bot

sogou web spider
sogou spider

checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)

mauibot
lightspeedsystemscrawler
abonti
aboundex
addthis.com
afd-verbotsverfahren
aibot
aihitbot
aipbot
alexibot
alligator
allsubmitter
alphabot
anarchie
apexoo
aspseek
asterias
attach
autoemailspider
backdoorbot
backlink-ceck
backlink-check
backlinkcrawler
backstreet
backweb
badass
bandit
barkrowler
batchftp
battleztar bazinga
bbbike
bdcbot
bdfetch
bender
berlin-fu-cow
betabot
bigfoot
bitacle
bl.uk_lddc_bot
black hole
blackboard
blackwidow
blow
blowfish
boardreader
bolt
botalot
brandprotect
brandwatch
bubing
buddy
builtbottough
builtwith
bullseye
bunnyslippers
buzzsumo
calculon
careerbot
catexplorador
cazoodlebot
ccbot
cegbfeieh
cheesebot
cherrypicker
chinaclaw
chlooe
claritybot
cliqzbot
cloud mapping
cloudservermarketspider
coccoc
coccocbot-web
cogentbot
cognitiveseo
collector
com.plumanalytics
copier
copyrightcheck
copyscape
cosmos
craftbot
crawler.feedback
crawler4j
crazywebcrawler
crazywebcrawler-spider
crescent
crystalsemanticsbot
cshttp
curious
custo
dblbot
demandbase-bot
demon
deusu
devil
digincore
digitalpebble
diibot
disco
discobot
discoverybot
dittospyder
dnyzbot
databasedrivermysqli
doc
domain re-animator bot
domainappender
domainstatsbot
download ninja
download wonder
dragonfly
drip
dts agent
easydl
ebingbong
ecatch
eccp
ecxi
eirgrabber
email siphon
email wolf
erocrawler
evc-batch
evil
exabot
express webpictures
extlinksbot
extractor
extractorpro
extreme picture finder
eyenetie
ezooms
fdm
femtosearchbot
fetch
fimap
flamingo_searchengine
flashget
flunky
foobot
fr-crawler
freeuploader
frontpage
fyrebot
g-i-g-a-b-o-t
galaxybot
garlikcrawler
genieo
germcrawler
getintent
getright
getweb
gigablast
gigabot
go!zilla
go-ahead-got-it
gotit
gozilla
gptbot
grabber
grabnet
grafula
grapefx
gridbot
haansoft
harvest
havij
headmasterseo
heritrix
hloader
hmview
humanlinks
hybridbot
iblog
iccrawler
id-search
idbot
ilsebot
image fetch
image sucker
implisensebot
indeedbot
indy library
infonavirobot
infotekies
instabid
integromedb
intelliseek
interget
internet ninja
internetseer
internetvista monitor
ips-agent
iria
irlbot
iskanie
istellabot
jamesbot
jbrofuzz
jennybot
jetcar
jikespider
jobboersebot
jobs.de-robot
joc web spider
joomla
jorgee
justview
jyxobot
k2spider
kenjin spider
keyword density
kozmosbot
kraken
lanshanbot
larbin
lcc
leechftp
leechget
lexibot
lexxebot
lftp
libweb
libwhisker
likse
linguee
linkdexbot
linkextractorpro
linko
linkpadbot
linkscan
linksmanager
linkstats
linkwalker
linqiametadatadownloaderbot
linqiarssbot
linqiascrapebot
lipperhey
lipperhey-kaus-australis
litemage_walker
lmspider
lnspiderguy
ltx71
lwp-trivial
mag-net
magnet
magpie-crawler
markmonitor
markwatch
mass downloader
masscan
mata hari
meanpathbot
mediawords
metajobbot
mfc_tear_sample
midown tool
miixpc
mindupbot
mister pix
mojeek
morfeus fucking scanner
mr.4x3
ms web services client protocol
msfrontpage
msiecrawler
msrabot
muhstik-scan
musobot
name intelligence
nameprotect
navroad
nearsite
needle
nerdbynature.bot
net vampire
netants
netcraft
netestate ne crawler
netlyzer
netmechanic
netseer crawler
netspider
nettrack
netvibes
netzip
nextgensearchbot
nibbler
nicerspro
niki-bot
nimblecrawler
ninja
npbot
nutch
obot
octopus
offline explorer
offline navigator
openfind
openhosebot
openlinkprofiler
openvas
orangebot
orangespider
orthogaffe
outclicksbot
outfoxbot
page analyzer
page scorer
pageanalyzer
pagegrabber
pagescorer
panscient
papa foto
paperlibot
pavuk
pcbrowser
peoplepal
phpcrawl
pi-monster
picscout
picsearch
picturefinder
pimonster
pixray
pixray-seeker
pleasecrawl
plista
plukkie
plumanalytics
pockey
poe-component-client-http
probethenet
propowerbot
prowebwalker
prtgcloudbot
psbot
pump
pxbroker
pycurl
queryn metasearch
queryseekerspider
quick-crawler
r6_commentreader
rankactive
rankactivelinkbot
rankflex
rankingbot
rankingbot2
rankivabot
ranksonicbot
rankurbot
realdownload
reaper
rebelmouse
recorder
redesscrapy
reget
repomonkey
ripper
rocketcrawler
rogerbot
safednsbot
salesintelligent
sbider
scan.lol
scanalert
scanbot
screenerbot
searchestate
searchmetricsbot
semager
sentibot
seodiver
seokicks
seokicks-robot
seolyticscrawler
seomoz
seoprofiler
seoscanners
seoscanners.net
seostats
sexsearcher
sg-orbiter
shodan
siphon
sistrix
sistrix crawler
site sucker
sitebeam
sitebot
sitecheck.internetseer.com
siteexplorer
siteimprove
sitelockspider
sitesnagger
sitesucker
sitevigil
slysearch
smartdownload
smtbot
snake
snapbot
snoopy
socialrankiobot
sosospider
sottopop
spacebison
spammen
spankbot
spanner
spiderbot
spinn3r
sputnikbot
sqlworm
sqworm
steeler
stripper
sucker
sucuri
superbot
superhttp
surfbot
surveybot
suzuran
swiftbot
sysscan
szukacz
t0phackteam
t8abot
takeout
teleport
teleportpro
telesoft
telesphoreo
telesphorep
the intraformant
thenomad
thumbsniper
tighttwatbot
titan
toata
toweyabot
trendiction
trendiction.com
trendiction.de
trendictionbot
trovitbot
true_robot
turingos
turnitin
turnitinbot
twengabot
twice
typhoeus
ubicrawler
um-ic
unisterbot
urly warning
urly.warning
vacuum
vagabondo
vb project
vci
vericitecrawler
vidiblescraper
virusdie
voideye
voil
voltron
wallpapers
wallpapershd
wasalive-bot
wbsearchbot
web auto
web collage
web enhancer
web fetch
web fuck
web pix
web sauger
web sucker
webalta
webauto
webbandit
webcollage
webdav
webenhancer
webfetch
webfuck
webgo is
webimagecollector
webleacher
webmasterworldforumbot
webpix
webreaper
websauger
webshag
website quester
websitequester
webster
websucker
webwhacker
wesee
wesee:search
whack
whacker
who.is bot
widow
wiseguys robot
wonderbot
woobot
wotbox
wwwoffle
x09mozilla
x22mozilla
xaldon webspider
xaldon_webspider
xenu
yasni
youdaobot
zade
zao
zauba
zauba.io
zealbot
zermelo
zeus
zgrab
zitebot
zmeu
zumbot
zyborg

Other Records

Comments

• Bad Search Engines
• Bing broken preview bot (no proper js support)
• User-agent: BingPreview
• Disallow: /
• Haosou 360 / Qihoo 360 [China] (http://www.so.com)
• Baidu [China] (http://www.baidu.com/search/spider.html)
• Mail.ru [Rusia] (http://go.mail.ru/help/robots)
• MJ12Bot [UK] (https://mj12bot.com/)
• Qwant.com [Francia] (https://www.qwant.com)
• Sogou [China] (http://www.sogou.com/docs/help/webmasters.htm)
• Yandex [Rusia] (https://yandex.com/support/webmaster/controlling-robot/robots-txt.html)
• User-agent: Yandex
• Disallow:/
• Block Commercial Bots
• Adbeat ads
• ADmantX [Alemania] (http://www.admantx.com/service-fetcher.html)
• Ahrefs [Singapur] (https://ahrefs.com/es/robot)
• Alexa (Amazon) [USA] (http://www.alexa.com/site/help/webmasters)
• Awario [Bielorusia] (https://awario.com/bots.html)
• webmeup.com [Bielorusia] (http://webmeup-crawler.com)
• builtwith.com [Australia] (https://builtwith.com)
• domaincrawler.com [Suecia] (http://www.domaincrawler.com)
• Moz / opensiteexplorer.org [USA] (https://moz.com/researchtools/ose/dotbot)
• Grapeshot (Oracle) [USA] (https://www.grapeshot.com/crawler)
• MegaIndex.ru [Rusia] (https://megaindex.com/crawler)
• Pinterest (Facebook) [USA] (https://help.pinterest.com/es/articles/about-pinterest-crawler)
• comScore [USA] (https://www.comscore.com/Web-Crawler)
• Semrush [USA] (http://www.semrush.com/bot.html)
• Serpstat [Ucrania] (https://serpstatbot.com/)
• seznam.cz [Republica Checa] (http://napoveda.seznam.cz/en/seznambot-intro)
• SiteExplorer.info [Holanda] (http://siteexplorer.info/Backlink-Checker-Spider)
• OpenLinkProfiler.org [USA] (http://OpenLinkProfiler.org/bot)
• velen.io [] (https://velen.io/)
• Sistrix [Alemania] (https://www.sistrix.com/)
• Xovi.de [Alemania] (http://www.xovibot.net)
• Yisou.com [China] (http://www.yisou.com)
• Cloud System Networks [USA] (http://cloudsystemnetworks.com/)
• Sogou [] (http://www.sogou.com/)
• Other unfriendly bots

Warnings

• 4 invalid lines.