sci.news
robots.txt

Groups

googlebot

googlebot-image

mediapartners-google/2.1

mediapartners-google*

msnbot

yahoo-mmcrawler

yahoo-blogs/v3.9

gigabot

robozilla

nutch

upday

domain re-animator bot

teleport

teleportpro

emailcollector

emailsiphon

webbandit

webzip

webreaper

webstripper

web downloader

webcopier

offline explorer pro

httrack website copier

offline commander

leech

websnake

blackwidow

http weazel

mj12bot
wesee
grapeshotcrawler
addthis.com
yandexbot
mail.ru bot
changedetection
exabot
genieo web filter
dotbot
naverbot
proximic
omgilibot
magpie-crawler
seoengbot
aihitbot
ahrefsbot
easouspider
woko
sistrix
spinn3r
flipboardproxy
blexbot
sogou spider
seznambot
searchmetricsbot
shopwiki
careerbot
wotbox
backlinkcrawler
baiduspider
vagabondo
coccoc
thumbshots-de-bot
pinterest
metajobbot
seegnifybot
nalezenczbot
spbot
ichiro
a6-indexer
umbot
openwebspider
yacybot
daumoa
aboundexbot
rogerbot
netseer
gocrawl
mojeekbot
trendictionbot
zumbot
showyoubot
istellabot
idmarch
linkdexbot
bixocrawler
browsershots
kraken
bitlybot
lipperhey spider
infohelfer
integromedb
crawler4j
jyxobot
loadimpactpageanalyzer
meanpathbot
antbot
askquickly
turnitinbot
bdcbot
stingbot
psbot
netcraftsurveyagent
urlappendbot
blekkobot
semrushbot
wbsearchbot
webcorp
bubing
seokicks-robot
motoricerca-robots.txt-checker
metageneratorcrawler
socialbm_bot
ccbot
hubspot connect
icjobs
company news search engine
x28-job-bot
everyonesocialbot
seobility
ezooms
symfony spider
iframely
nextgensearchbot
krowler
netestate crawler
twingly recon
robots_tester
facebookexternalhit
obot
arachnophilia
ecommercebot
alexabot
emefgebot
uaslinkchecker
nuhk
panscient web crawler
najdi.si
securityresearchbot
cloudservermarketspider
yyspider
unisterbot
icc-crawler
peeplo screenshot bot
steeler
nekstbot
loadtimebot
spiderling
webinatorbot
cliqzbot
leikibot
aboutusbot
tineye
musobot
search.kumkie.com
nigma.ru
compspybot
seocheckbot
hawkreader
percolatecrawler
butterfly
plukkie
webthumbnail
falconsbot
ssl-crawler
thumbsniper
embedly
linguatools
backlink-check.de
adressendeutschland.de
xrl
ideelaborplagiaat
sitecondor
web-monitoring
vedma
parsijoo
garlikcrawler
fyberspider
classbot
zeerchbot
feedly
webcookies
linkedinbot
tomtom places company search
cloudflare-alwaysonline
readability
suggybot
catchbot
jabse.com crawler
woriobot
exb language crawler
kulturarw
brainbrubot
komodiabot
qualidator.com bot
ixebot
cms crawler
immediatenet thumbnails
shareaholicbot
yioopbot
qualidator.com siteanalyzer 1.0
qirina hurdler
begunadvertising
luminatebot
linkdex.com
curious george
fetch-guess
sbsearch
alexa site audit
arabot
amznkassocbot
speedy
hosttracker
cliqzbot
findlinks
ccresearchbot
semantifire
linkaider
zookabot
screenerbot crawler
webmastercoffee
paperlibot
queryseekerspider
crowsnest
unwindfetchor
metauri api
miadev
acoonbot
firmilybot
sosospider
openindexspider
metaheadersbot
strokebot
geliyoobot
bot-pge.chlooe.com
owncloud server crawler
cirrusexplorer
procogseobot
dlvr.it/1.0
open web analytics bot
ryzecrawler
discoverybot
crawler for netopian
admantx platform semantic analyzer
r6 bot
bl.uk_lddc_bot
linguee bot
solomonobot
grahambot
automattic analytics crawler
youdaobot
piplbot
flightdeckreportsbot
fastbot crawler
updownerbot
jikespider
nlnz_iaharvester2013
wsanalyzer
yodaobot
esribot
thumbshots.ru
blogpulse
bot.wsowner.com
wscheck.com
qseero
drupact
huaweisymantecspider
pagepeeker
hometags
facebookplatform
pixray-seeker
bdfetch
memonewsbot
procogbot
willybot
peerindex
job roboter spider
mlbot
webnl
peepowbot
semager
mia bot
heritrix
eurobot
dripfeedbot
whoismindbot
bad-neighborhood
hailoobot
akula
metamojicrawler
page2rss
easybib autocite
nerdbynature.bot
eventgurubot
quickobot
gonzo
bnf.fr_bot
uptimerobot
influencebot
msrbot
keyworddensityrobot
ronzoobot
scoutjet
twikle
swebot
radar-bot
dcpbot
castabot
imbot
edisterbot
wasalive-bot
accelobot
postpost
factbot
setoozbot
biwec
search17bot
lijit
just-crawler
apercite
pmoz.info odp link checker
lemurwebcrawler
covario-ids
holmes
rankurbot
envolk
ask jeeves/teoma
lexxebot
stackrambler
abrave spider
evrinid
arachnode.net
camontspider
wikiwix-bot
nymesis
trendictionbot
sitedomain-bot
seodat
sygolbot
snapbot
opencalaissemanticproxy
zookabot
cligoorobot
cityreview
nworm
sbider
dot tk - spider
euripbot
parchbot
peew
yrspider
urlfilebot (urlbot)
gaisbot
watchmouse
tagoobot
webwatch/robot_txtchecker
urlfan-bot
statoolsbot
page_verifier
sslbot
sai crawler
domaindb
linkwalker
wmcai_robot
voyager
copyright sheriff
ocelli
twiceler
amibot
abby
netresearchserver
videosurf_bot
xml sitemaps generator
blinkacrawler
nodestackbot
pompos
taptubot
babaloospider
yaanb
girafabot
livedoor screenshot
ecairn-grabber
faubot
toread-crawler
setoozbot
metauri
l.webis
web-sniffer
fairshare
ruky-roboter
thumbshots-bot
botonparade
amagit.com
hatenascreenshot
holmesbot
dotsemantic
karneval-bot
hosttracker.com
aportworm
xmarksfetch
feedfinder/bloggz.se
corpuscrawler
willow internet crawler
orgbybot
gingercrawler
pingdom.com_bot
baypup
mp3bot
surphace scout
wikiofeedbot
szukacz
dblbot
thumbnail.cz robot
linguabot
gurujibot
charlotte
sanszbot
moba-crawler
heartrails_capture
surveybot
mnogosearch
smart.apnoti.com robot
topicbot
jadynavebot
osobot
webimages
winwebbot
scooter
scarlett
goforitbot
dkimrepbot
yanga
dns-digger-explorer
yowedobot
botmobi
fooooo_web_video_crawl
uptimedog
metaspinner/0.01
touche
rssmicro.com rss/atom feed robot
sniffrss
kalooga
feedcatbot
webrankspider
flatland industries web spider
dealgates bot
link valet online
shelob
technoratibot
flocke bot
followsite bot
visbot
livelapbot
semantic-visions.com crawler
scooperbot
buzzsumo
veooz
maxpointcrawler
grapeshotcrawler
freewebmonitoring sitechecker
toutiaospider
trendictionbot
netseer crawler
tweetmemebot
semrushbot
sogou web spider
newsharecounts.com

Other Records

Comments

• This is a list of web-site copiers
• Other robots

Warnings

• 8 invalid lines.