hrhresourcecenter.org
robots.txt

Groups

alexibot
aqua_products
bpimagewalker/2.0
backdoorbot
backdoorbot/1.0
baidu
baiduspider
baiduspider-ads
baiduspider-cpro
baiduspider-favo
baiduspider-image
baiduspider-news
baiduspider-video
black.hole
blackwidow
blekkobot
blowfish
blowfish/1.0
bookmark search tool
bot mailto:craftbot@yahoo.com
botalot
botrighthere
builtbottough
bullseye
bullseye/1.0
bunnyslippers
cegbfeieh
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
copernic
copyrightcheck
crescent
crescent internet toolpak http ole control v.1.0
custo
disco
disco pump 3.0
disco pump 3.2
discofinder
dittospyder
download demon
download demon/3.2.0.8
download demon/3.5.0.11
earlyedd
earlyedd http://earlyedd.com
eirgrabber
emailcollector
emailsiphon
emailwolf
erocrawler
express webpictures
express webpictures (www.express-soft.com)
extractorpro
eyenetie
fairad client
flaming attackbot
flashget
flashget webwasher 3.2
foobot
frontpage
frontpage [nc,or]
gaisbot
getright
getright/2.11
getright/3.1
getright/3.2
getright/3.3
getright/3.3.3
getright/3.3.4
getright/4.0.0
getright/4.1.0
getright/4.1.1
getright/4.1.2
getright/4.2
getright/4.2b
getright/4.2b (portuguxeas)
getright/4.2c
getright/4.3
getright/4.5
getright/4.5a
getright/4.5b
getright/4.5b1
getright/4.5b2
getright/4.5b3
getright/4.5b6
getright/4.5b7
getright/4.5c
getright/4.5d
getright/4.5e
getright/5.0beta1
getright/5.0beta2
getweb!
go!zilla
go!zilla (www.gozilla.com)
go!zilla 3.3 (www.gozilla.com)
go!zilla 3.5 (www.gozilla.com)
go-ahead-got-it
grabnet
grafula
hmview
httrack
httrack 3.0
httrack 3.0x
httrack [nc,or]
harvest
harvest/1.5
image stripper
image sucker
indy library
indy library [nc,or]
infonavirobot
interget
internet ninja
internet ninja 4.0
internet ninja 5.0
internet ninja 6.0
iron33/1.0.2
joc web spider
js-kit url resolver
js-kit url resolver, http://js-kit.com/
java/1.6.0_07
jennybot
jetcar
kenjin spider
kenjin.spider
keyword density/0.9
keyword.density
lnspiderguy
leechftp
lexibot
linkscan/8.1a unix
linkscan/8.1a.unix
linkwalker
linkextractorpro
midown tool
miixpc
miixpc/4.2
mj12bot
msiecrawler
mass downloader
mass downloader/2.2
mata hari
mata.hari
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
microsoft.url
mister pix
mister pix version.dll
mister pix ii 2.01
mister pix ii 2.02a
mister.pix
nicerspro
ning/1.0
npbot
naverbot
navroad
nearsite
net vampire
net vampire/3.0
netants
netants/1.10
netants/1.23
netants/1.24
netants/1.25
netmechanic
netspider
netzip
netzip downloader 1.0 win32(nov 12 1998)
netzip-downloader/1.0.62 (win32; dec 7 1998)
netzippy
netzippy+(http:/www.innerprise.net/usp-spider.asp)
octopus
offline explorer
offline explorer/1.2
offline explorer/1.4
offline explorer/1.6
offline explorer/1.7
offline explorer/1.9
offline explorer/2.0
offline explorer/2.1
offline explorer/2.3
offline explorer/2.4
offline explorer/2.5
offline navigator
offline.explorer
openwebspider
openbot
openfind
openfind data gatherer
oracle ultra search
outfoxbot/0.5
pagegrabber
papa foto
perman
postrank/2.0
postrank/2.0 (postrank.com)
propowerbot/2.14
prowebwalker
pycurl/7.18.2
python-urllib
queryn metasearch
queryn.metasearch
rma
radiation retriever 1.1
reget
realdownload
realdownload/4.0.0.40
realdownload/4.0.0.41
realdownload/4.0.0.42
repomonkey
repomonkey bait & tackle/v1.01
semrushbot
semrushbot/0.96.2
sitesnagger
slysearch
smartdownload
smartdownload/1.2.76 (win32; apr 1 1999)
smartdownload/1.2.77 (win32; aug 17 1999)
smartdownload/1.2.77 (win32; feb 1 2000)
smartdownload/1.2.77 (win32; jun 19 2001)
socialmedia bot/1.5;
spankbot
sqworm/2.9.85-beta
sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux
sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux)
summify/1.0.1
superbot
superbot/3.0 (win32)
superbot/3.1 (win32)
superhttp
superhttp/1.0
superfeedr bot/2.0
surfbot
szukacz/1.4
teleport
teleport pro
teleport pro/1.29
teleport pro/1.29.1590
teleport pro/1.29.1634
teleport pro/1.29.1718
teleport pro/1.29.1820
teleport pro/1.29.1847
teleportpro
telesoft
the intraformant
the.intraformant
thenomad
tighttwatbot
titan
true_robot
true_robot/1.0
turnitinbot
turnitinbot/1.5
tweetmemebot
twitterbot/0.1
typhoeus - https://github.com/typhoeus/typhoeus
url control
url_spider_pro
urly warning
urly.warning
unwindfetchor/1.0
unwindfetchor/1.0 (+http://www.gnip.com/)
vci
vci webviewer vci webviewer win32
voideye
voyager/1.0
www-collector-e
wwwoffle
web image collector
web sucker
web.image.collector
webauto
webauto/3.40 (win98; i)
webbandit
webbandit/3.50
webcapture 2.0
webcopier
webcopier v.2.2
webcopier v2.5
webcopier v2.6
webcopier v2.7a
webcopier v2.8
webcopier v3.0
webcopier v3.0.1
webcopier v3.2
webcopier v3.2a
webemailextrac
webenhancer
webfetch
webgo is
webleacher
webreaper
webreaper [info@webreaper.net]
webreaper [webreaper@otway.com]
webreaper v9.1 - www.otway.com/webreaper
webreaper v9.7 - www.webreaper.net
webreaper v9.8 - www.webreaper.net
webreaper vwebreaper v7.3 - www,otway.com/webreaper
websauger
websauger 1.20b
websauger 1.20j
websauger 1.20k
webstripper
webstripper/2.03
webstripper/2.10
webstripper/2.12
webstripper/2.13
webstripper/2.15
webstripper/2.16
webstripper/2.19
webwhacker
webzip/2.75 (http:/www.spidersoft.com)
webzip/3.65 (http:/www.spidersoft.com)
webzip/3.80 (http:/www.spidersoft.com)
webzip/4.0 (http:/www.spidersoft.com)
webzip/4.1 (http:/www.spidersoft.com)
webzip/4.21
webzip/4.21 (http:/www.spidersoft.com)
webzip/5.0
webzip/5.0 (http:/www.spidersoft.com)
webzip/5.0 pr1 (http:/www.spidersoft.com)
webzip
webzip/4.0
webmasterworldforumbot
website quester
website quester - www.asona.org
website quester - www.esalesbiz.com/extra/
website extractor
website extractor (http:/www.asona.org)
website.quester
webster pro
webster.pro
widow
xaldon webspider
xaldon webspider 2.5.b3
xenu's
xenu's link sleuth 1.1c
yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp
yandexbot/3.0
yandexwebmaster/2.0
yeti
youdaobot
zeus
zeus 11389 webster pro v2.9 win32
zeus 11652 webster pro v2.9 win32
zeus 18018 webster pro v2.9 win32
zeus 26378 webster pro v2.9 win32
zeus 30747 webster pro v2.9 win32
zeus 32297 webster pro v2.9 win32
zeus 39206 webster pro v2.9 win32
zeus 41641 webster pro v2.9 win32
zeus 44238 webster pro v2.9 win32
zeus 51070 webster pro v2.9 win32
zeus 51674 webster pro v2.9 win32
zeus 51837 webster pro v2.9 win32
zeus 63567 webster pro v2.9 win32
zeus 6694 webster pro v2.9 win32
zeus 71129 webster pro v2.9 win32
zeus 82016 webster pro v2.9 win32
zeus 82900 webster pro v2.9 win32
zeus 84842 webster pro v2.9 win32
zeus 90872 webster pro v2.9 win32
zeus 94934 webster pro v2.9 win32
zeus 95245 webster pro v2.9 win32
zeus 95351 webster pro v2.9 win32
zeus 97371 webster pro v2.9 win32
zeus link scout
asterias
b2w/0.1
bitlybot
brainbot/1.0
brainbot/1.0 (digitalbrain@brainstreammedia @digitalbrain
complex_network_group
cosmos
dotbot
ecatch
ecatch/3.0
hloader
httplib
humanlinks
ichiro
ichiro/3.0
ichiro/4.0
ichiro/5.0
kame-rt
kame-rt (support@backtype.com
kame-rt (support@backtype.com)
larbin
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
larbin_2.6.2 (kabura@sushi.com)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (listonatccdotgatechdotedu)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 kabura@sushi.com
larbin_2.6.2 larbin2.6.2@unspecified.mail
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 listonatccdotgatechdotedu
larbin_2.6.2 vitalbox1@hotmail.com
libweb/clshttp
lwp-trivial
lwp-trivial/1.34
moget
moget/2.1
pavuk
pcbrowser
proximic
psbot
searchpreview
sogou spider
spanner
suzuran
takeout
tocrawl/urldispatcher
turingos
webfetch/2.1.0

*

Other Records

ahrefsbot

Comments

• $Id: robots.txt 11 2011-01-14 15:38:10Z pcardullo $
• robots.txt
• This file is to prevent the crawling and indexing of certain parts
• of your site by web crawlers and spiders run by sites like Yahoo!
• and Google. By telling these "robots" where not to go on your site,
• you save bandwidth and server resources.
• This file will be ignored unless it is at the root of your host:
• Used: http://example.com/robots.txt
• Ignored: http://example.com/site/robots.txt
• For more information about the robots.txt standard, see:
• http://www.robotstxt.org/wc/robots.html
• For syntax checking, see:
• http://www.sxw.org.uk/computing/robots/check.html
• Bots Galore
• Directories
• Disallow: /misc/
• Files
• Paths (clean URLs)
• Paths (no clean URLs)
• Block ahrefsbot

Warnings

• 1 invalid line.