friedsocialworker.com
robots.txt

Groups

abot
abot/0.1
abot@abot.com
aboutusbot
aipbot
aipbot/1.0
aipbot/2
aipbot/2-beta
aipbot@aipbot.com
aipbot dev
alkalinebot
alkalinebot/1.3
alkalinebot/1.4
alkalinebot/1.4.0326.0 rtm
almaden.ibm.com
aol sureseeker search plugin
aport
archive.org_bot
archive.org_bot/1.6.0
assort
assort/0.10
associative sort
b2w
b2w/0.1
baidu
baiduspider
baiduspider
baiduspider+
become
becomebot
becomebot/2.x
becomebot/2.3
becomebot/3.0
beijingcrawler
blogsearchbot
blogsearchbot-martin-x
blogsearchbot-pumpkin-2
boitho.com-dc/0.81
boxseabot
charlotte
charlotte/1.0b
charlotte@betaspider.com
checkbot
checkbot/x.xx lwp/5.x
collage.cgi
collage.cgi/1.82
convera internet spider
convera internet spider v6.x
convera internet spider v6.9
convera internet spider v6.9
converacrawler
converacrawler/0.2
converacrawler/0.9d
converamultimediacrawler
converamultimediacrawler/0.1
cowbot
cowbot-0.1
cowbot-0.1.x
crawlconvera
depspid
dev-spider2.ercko.com/1.3b
dev-spider2.searchspider.com/1.3b
dc/0.81
diamondbot
digger
digger/1.0 jdk/1.3.0rc3
digi-rssbot
dittospyder
dloader
dloader(naverrobot)/1.0
dloader/1.0
easydl
easydl/3.04
elsop
exabot
exabot/2.0
exabot/3.0
exabot-images
exabot-images/1.0
findexa crawler
fast freshcrawler 6
fi crawler
geniebot
geniebot wgao
gigabot
gigabot/2.0
girafa
grub
grub-client
googlebot-image
hl_ftien_spider
hl_ftien_spider_v1.1
hakulaite
hakulaite/0.1
healthline
heritrix
ia_archiver
ia_archiver/1.6
ia_archiver-web.archive.org
iaarchiver-1.0
ichiro
ichiro/2.0
ichiro@nttr.co.jp
infoconveracrawler
ipaddressguidebot
ipaddressguidebot/1.1
ip2phrasebot
ip2phrasebot/1.0
irlbot
irlbot/1.0
kaklebot
kakle-spider
kakle-spider/0.1
krugle
krugle/krugle
krugle web crawler
ksibot
ksibot/8.0d
lanshanbot
lanshanbot/1.0
larbin
larbin_2.1.1 larbin2.1.1
larbin_2.1.1 larbin2.1.1@somewhere.com
larbin_2.2.0
larbin_2.2.1_de_viennot
larbin_2.2.2
larbin_2.2.2_guillaume
larbin_2.6.0
larbin_2.6.1
larbin_2.6.2
larbin_2.6.3
larbin_2.6.3_for_
larbin_2.6_basileocaml
larbin_devel
larbin-experimental
linkwalker
linkscan
linksmanager link checker bot
linksmanager.com
ljseek picture-bot
ljseek picture-bot/1.0
metager-linkchecker
metagerbot
metagerbot/0.8-dev
mj12bot
mj12bot/v1.0.7
mj12bot/1.0.7
mj12bot/v1.0.8
moget
moget@goo.ne.jp
mogimogi
mogimogi/1.0
mqbot
mqbot metaquerier
mqbot metaquerier.cs.uiuc.edu/crawler
msrbot
metaexplorer
myfamilybot
myfamilybot/1.0
nabot
nabot_1.0
nabot/5.0
naverbot
naverbot-1.0
naverbot/1.0
naverbot_dloader/1.5
naverrobot
nhnbot@naver.com
nhnbot
nhn corp.
nhn corp. / +82-2-3011-1954 / nhnbot@naver.com
nextgensearchbot
nextgensearchbot 1
nicebot
noxtrumbot
noxtrumbot/1.0
np
npbot
npbot@nameprotect.com
np/0.1
npbot-1/2.0
obot
obot ((compatible;win32))
pulsebot (pulse web miner)
pulsebot
pulse web miner
python
python-urllib
python-urllib/1.1x
python-urllib/2.0a1
nutchorg
nutch
nutchcvs
nutchcvs/0.05
nutchcvs/0.06-dev
nutchcvs/0.7.1
nutchcvs/0.7.2
nutchcvs/0.8-dev
nutch/0.8+
http://24.177.134.x
http://64.5.245.11
http://64.5.245.11/faq/faq.html
http://64.124.122.252
http://64.124.122.252/feedback.html
http://66.234.139.194
http://207.241.225.2xx
http://www.abot.com
http://www.aipbot.com
http://alkaline.vestris.com/
http://www.almaden.ibm.com/cs/crawler
http://www.almaden.ibm.com/robot
http://www.aport.ru
http://www.archive.org
http://www.authoritativeweb.com
http://www.authoritativeweb.com/crawl
http://www.baidu.com
http://www.baidu.com/search/spider.htm
http://www.bb2.net
http://www.become.com
http://www.become.com/site_owners.html
http://www.become.com/webmasters.html
http://www.betaspider.com
http://www.boitho.com/dcbot.html
http://www.cobion.com
http://www.convera.com
http://cosco.hiit.fi/search/
http://www.downloadaccelerator.com
http://ego.ms.mff.cuni.cz/
http://www.elsop.com
http://www.exabot.com
http://www.findexa.no/gulesider/article26548.ece
http://www.genieknows.com
http://www.gigablast.com/spider.html
http://help.goo.ne.jp/door/crawler.html
http://help.goo.ne.jp/
http://www.goo.ne.jp/
http://www.healthline.com
http://www.indyproject.org/
http://irl.cs.tamu.edu/crawler
http://www.kakle.com
http://www.kakle.com/0.1
http:// www.kakle.com/bot.html
http://keywen.com/encyclopedia/bot
http://corp.krugle.com/crawler/info.html
http://www.krugle.com
http://www.krugle.com/crawler/info.html
webcrawler@krugle.com
http://www.linksmanager.com
http://linksmanager.com/linkchecker.html
http://www.loc.gov/minerva/crawl.html
http://www.ljpic.com
http://lucene.apache.org/nutch/bot.html
http://www.macedition.com
http://majestic12.co.uk/bot.php
http://www.metager.de
http://metaquerier.cs.uiuc.edu/crawler/
http://www.myfamilyinc.com
http://www.nameprotect.com
http://www.nameprotect.com/botinfo.html
http://www.naver.co.jp/
http://www.naver.com
http://www.netidea.it
http://www.netiq.com/
http://www.netiq.com/webtrends/default.asp
http://www.netwu.com/
http://www.netwu.com/webpix/
http://www.noxtrum.com
http://www.nutch.org
http://www.omni-explorer.com
http://pauillac.inria.fr/~ailleret/prog/larbin/
http://www.pediasearch.com
http://www.python.org/
http://research.microsoft.com/research/sv/msrbot/
http://www.sensis.com.au/
http://www.schibstedsok.no/bot/
http://www.searchscout.com
http://www.seekbot.net/bot.html
http://www.shopwiki.com/wiki/help:bot
http://www.snap.com
http://home.snafu.de/tilman/xenulink.html
http://www.snafu.de/
sougou_spider@sohu-rd.com
http://www.sohu-rd.com
http://www.speedbit.com
http://www.sproose.com/
http://www.sproose.com/bot.html
http://www.sureseeker.com
http://www.sygol.com/
http://www.sygol.net
http://www.terrawiz.com/
http://www.umechando.com
http://www.umechando.com/webex/
http://www.webaroo.com
http://www.webcollage.com
http://webstripper.net
http://webstripper.net/index.html
http://www.wisenut.com/robot
http://www.whois.sc
http://wume.cse.lehigh.edu/~xiq204/crawler/
http://www.vestris.com/
http://www.zdnet.com
mozilla
mozilla/5.0
nimblecrawler
nimblecrawler 2.0.1
np
np/0.1
obot
omniexplorer_bot
omniexplorer_bot/6.48
omniexplorer_bot/6.57
omniexplorer_bot/6.65a
omniexplorer_bot/6.66
omniexplorer_bot/6.68
omniexplorer_bot/6.70
pediasearch
pediasearch.com crawler
pigeonbot
pigeonbot1.0 beta
psbot
python-urllib/1.15
rufusbot
rufus web miner
search_comments\at\sensis\dot\com\dot\au
sensis web crawler
sensis.com.au web crawler
schibstedsokbot
seekbot
seekbot/1.0
shopwiki
shopwiki/1.0
sitesnagger
snapbot
snapbot/1.0
snap.com beta crawler
snap.com beta crawler v0
sogou spider
sogou pic spider
sogou pic spider/3.0
sohu agent
sohu-search
spider indexer
spider indexer beta2
sproose
sproose/0.1-alpha
sproose/1.0beta
sproose bot
crawler@sproose.com
surveybot
surveybot/2.2
surveybot/2.3
sygol
sygolbot
sygolbot http://www.sygol.net
sygolbot http://www.sygol.com
teoma
terrawizbot
terrawizbot/1.0
tmcrawler
vestris
voila
voila.fr
vscooter
webaroobot
webaroo bot
webaroobot/rufusbot
webcollage
webcollage/1.xx
webcollage syndicator
webpix
webpix 1.0
webpix 1.0 (www.netwu.com)
web image collector
website explorer
website explorer/0.9.x.x
web stripper
webstripper
webstripper/2.0x
webstripper/2.xx
webtrends
webtrends/3.0 (winnt)
webzip
wespe.de
whois source
worldindexer
wume_crawler
wume_crawler/1.1
www.fi crawler
xenu's link sleuth
xenu's link sleuth 1.x[a-z]

googlebot

slurp

*

Warnings

• 1 invalid line.