dcs1225.com
robots.txt

Groups

mozilla/3.0 (compatible;miner;mailto:miner@miner.com.br)

webferret

due to a deficiency in java it's not currently possible

no

arachnophilia

architextspider

aspider/0.09

auresys/1.0

backrub/*.*

big brother

blackwidow

bspider/1.0 libwww-perl/0.40

cactvs chemistry spider

digimarc cgireader/1.0

checkbot/x.xx lwp/5.x

cmc/0.01

combine/0.0

conceptbot/0.3

crescent internet toolpak http ole control v.1.0

root/0.1

cs-hkust-indexserver/1.0

cyberspyder/2.1

deweb/1.01

dragonbot/1.0 libwww/5.0

eit-link-verifier-robot/0.2

emacs-w3/v[0-9\.]+

emailsiphon

emc spider

explorersearch

explorer

extractorpro

felixide/1.0

hazel's ferret web hopper,

esirover v1.0

fido/0.9 harvest/1.4.pl2

hämähäkki/0.2

kit-fireball/2.0 libwww/5.0a

fish-search-robot

mozilla/2.0 (compatible fouineur v2.0;

robot du crim 1.0a

freecrawl

funnelweb-1.0

gcreep/1.0

geturl.rexx v1.05

golem/1.1

gromit/1.0

gulliver/1.1

yes

aitcsrobot/1.1

wired-digital-newsbot/1.5

htdig/3.0b3

htmlgobble v2.2

no

ibm_planetwide,

gestalticonoclast/1.0 libwww-fm/2.17

ingrid/0.1

incywincy/1.0b1

informant

infoseek robot 1.0

infoseek sidewinder

infospiders/0.1

inspectorwww/1.0

israelisearch/1.0

jcrawler/0.2

jeeves v0.05alpha (perl, lwp, lglb@doc.ic.ac.uk)

jobot/0.1alpha libwww-perl/4.0

joebot,

jubiirobot

jumpstation

katipo/1.0

kdd-explorer/0.1

ko_yappo_robot/1.0.4(http://yappo.com/info/robot.html)

labelgrab/1.1

linkwalker

logo.gif crawler

lycos/x.x

lycos_spider_(t-rex)

magpie/1.0

mediafox/x.y

merzscope

nec-meshexplorer

momspider/1.00 libwww-perl/0.40

monster/vx.x.x -$type ($ostype)

motor/0.2

muscatferret

mwdsearch/0.1

netcarta cyberpilot pro

netmechanic

netscoop/1.0 libwww/5.0a

nhsewalker/3.0

nomad-v2.x

northstar

occam/1.0

hku www robot,

orbsearch/1.0

packrat/1.0

patric/0.01a

peregrinator-mathematics/0.7

duppies

pioneer

pgp-ka/1.2

resume robot

road runner: imagescape robot (lim@cs.leidenuniv.nl)

robbie/0.1

computingsite robi/1.0 (robi@computingsite.com)

roverbot

safetynet robot 0.1,

scooter/1.0

not available

senrigan/xxxxxx

sg-scout

shai'hulud

simbot/1.0

open text site crawler v1.0

sitetech-rover

slurp/2.0

esismartspider/2.0

snooper/b97_01

solbot/1.0 lwp/5.07

spanner/1.0 (linux 2.0.27 i586)

no

mozilla/3.0 (black widow v1.1.0; linux 2.0.27; dec 31

tarantula/1.0

tarspider

dlw3robot/x.y (in tclx by http://hplyot.obspm.fr/~dl/)

templeton/

titin/0.2

titan/0.1

ucsd-crawler

urlck/1.2.3

valkyrie/1.0 libwww-perl/0.40

victoria/1.0

vision-search/3.0'

vwbot_k/4.2

w3index

w3m2/x.xxx

wwwwanderer v3.0

webcopy/

webcrawler/3.0 robot libwww/5.0a

webfetcher/0.8,

weblayers/0.0

weblinker/0.0 libwww-perl/0.1

no

webmoose/0.0.0000

digimarc webreader/1.2

webs@recruit.co.jp

webvac/1.0

webwalk

webwalker/1.10

webwatch

wget/1.4.0

w3mir

no

wwwc/0.25 (win95)

none

xget/0.7

nederland.zoek

bizbot04 kirk.overleaf.com

happybot (gserver.kw.net)

californiabrownspider

ei*net/0.1 libwww/0.1

ibot/1.0 libwww-perl/0.40

merritt/1.0

statfetcher/1.0

teachersoft/1.0 libwww/2.17

www collector

processor/0.0alpha libwww-perl/0.20

wobot/1.0 from 206.214.202.45

libertech-rover www.libertech.com?

whowhere robot

iti spider

w3index

mycnnspider

summycrawler

ogspider

linklooker

cyberspyder (amant@www.cyberspyder.com)

slowbot

heraspider

surfbot