djsilver.com
robots.txt

Robots Exclusion Standard data for djsilver.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	djsilver.com
Base Domain	djsilver.com
Scan Status	Ok
Last Scan	2025-10-29T15:08:37+00:00
Next Scan	2025-11-05T15:08:37+00:00

Last Scan

Scanned	2025-10-29T15:08:37+00:00
URL	https://djsilver.com/robots.txt
Domain IPs	104.21.44.65, 172.67.196.113, 2606:4700:3034::6815:2c41, 2606:4700:3035::ac43:c471
Response IP	172.67.196.113
Found	Yes
Hash	2d3aa09b8f3204b9e601b65a389e87c9845bd3b9c15188ca631249fd68d7ea43
SimHash	c079ca9e7bd3

Groups

mediapartners-google*

Rule	Path
Disallow	/admin/
Disallow	/banner/
Disallow	/%21errorfiles/
Disallow	/lib/

Rule

Path

Disallow

/admin/

Disallow

/banner/

Disallow

/%21errorfiles/

Disallow

/lib/

*

Rule	Path
Disallow	/admin/
Disallow	/banner/
Disallow	/%21errorfiles/
Disallow	/lib/

Rule

Path

Disallow

/admin/

Disallow

/banner/

Disallow

/%21errorfiles/

Disallow

/lib/

dittospyder
vscooter

Rule	Path
Disallow	/

Rule

Path

Disallow

webferret

Rule	Path
Disallow

Rule

Path

Disallow

due to a deficiency in java it's not currently possible to set the user-agent.

Rule	Path
Disallow

Rule

Path

Disallow

aspider/0.09

Rule	Path
Disallow

Rule

Path

Disallow

auresys/1.0

Rule	Path
Disallow

Rule

Path

Disallow

backrub/.

Rule	Path
Disallow

Rule

Path

Disallow

blackwidow

Rule	Path
Disallow

Rule

Path

Disallow

cactvs chemistry spider

Rule	Path
Disallow

Rule

Path

Disallow

digimarc cgireader/1.0

Rule	Path
Disallow

Rule

Path

Disallow

checkbot/x.xx lwp/5.x

Rule	Path
Disallow

Rule

Path

Disallow

cmc/0.01

Rule	Path
Disallow

Rule

Path

Disallow

combine/0.0

Rule	Path
Disallow

Rule

Path

Disallow

root/0.1

Rule	Path
Disallow

Rule

Path

Disallow

cs-hkust-indexserver/1.0

Rule	Path
Disallow

Rule

Path

Disallow

emacs-w3/v[0-9\.]+

Rule	Path
Disallow

Rule

Path

Disallow

emailsiphon

Rule	Path
Disallow

Rule

Path

Disallow

emc spider

Rule	Path
Disallow

Rule

Path

Disallow

explorersearch

Rule	Path
Disallow

Rule

Path

Disallow

explorer

Rule	Path
Disallow

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow

Rule

Path

Disallow

felixide/1.0

Rule	Path
Disallow

Rule

Path

Disallow

hazel's ferret web hopper,

Rule	Path
Disallow

Rule

Path

Disallow

esirover v1.0

Rule	Path
Disallow

Rule

Path

Disallow

fido/0.9 harvest/1.4.pl2

Rule	Path
Disallow

Rule

Path

Disallow

hï¿½mï¿½hï¿½kki/0.2

Rule	Path
Disallow

Rule

Path

Disallow

kit-fireball/2.0 libwww/5.0a

Rule	Path
Disallow

Rule

Path

Disallow

fish-search-robot

Rule	Path
Disallow

Rule

Path

Disallow

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

Rule	Path
Disallow

Rule

Path

Disallow

robot du crim 1.0a

Rule	Path
Disallow

Rule

Path

Disallow

freecrawl

Rule	Path
Disallow

Rule

Path

Disallow

funnelweb-1.0

Rule	Path
Disallow

Rule

Path

Disallow

gcreep/1.0

Rule	Path
Disallow

Rule

Path

Disallow

geturl.rexx v1.05

Rule	Path
Disallow

Rule

Path

Disallow

golem/1.1

Rule	Path
Disallow

Rule

Path

Disallow

gromit/1.0

Rule	Path
Disallow

Rule

Path

Disallow

gulliver/1.1

Rule	Path
Disallow

Rule

Path

Disallow

yes

Rule	Path
Disallow

Rule

Path

Disallow

aitcsrobot/1.1

Rule	Path
Disallow

Rule

Path

Disallow

htdig/3.0b3

Rule	Path
Disallow

Rule

Path

Disallow

htmlgobble v2.2

Rule	Path
Disallow

Rule

Path

Disallow

gestalticonoclast/1.0 libwww-fm/2.17

Rule	Path
Disallow

Rule

Path

Disallow

ingrid/0.1

Rule	Path
Disallow

Rule

Path

Disallow

incywincy/1.0b1

Rule	Path
Disallow

Rule

Path

Disallow

informant

Rule	Path
Disallow

Rule

Path

Disallow

infospiders/0.1

Rule	Path
Disallow

Rule

Path

Disallow

inspectorwww/1.0 http://www.greenpac.com/inspectorwww.html

Rule	Path
Disallow

Rule

Path

Disallow

israelisearch/1.0

Rule

Path

Disallow

jcrawler/0.2

Rule

Path

Disallow

jeeves v0.05alpha (perl, lwp, lglb@doc.ic.ac.uk)

Rule

Path

Disallow

jobot/0.1alpha libwww-perl/4.0

Rule

Path

Disallow

joebot,

Rule

Path

Disallow

jubiirobot

Rule

Path

Disallow

jumpstation

Rule

Path

Disallow

katipo/1.0

Rule

Path

Disallow

kdd-explorer/0.1

Rule

Path

Disallow

labelgrab/1.1

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

logo.gif crawler

Rule

Path

Disallow

magpie/1.0

Rule

Path

Disallow

mediafox/x.y

Rule

Path

Disallow

merzscope

Rule

Path

Disallow

nec-meshexplorer

Rule

Path

Disallow

momspider/1.00 libwww-perl/0.40

Rule

Path

Disallow

monster/vx.x.x -$type ($ostype)

Rule

Path

Disallow

motor/0.2

Rule

Path

Disallow

muscatferret

Rule

Path

Disallow

mwdsearch/0.1

Rule

Path

Disallow

netcarta cyberpilot pro

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

netscoop/1.0 libwww/5.0a

Rule

Path

Disallow

nhsewalker/3.0

Rule

Path

Disallow

nomad-v2.x

Rule

Path

Disallow

northstar

Rule

Path

Disallow

occam/1.0

Rule

Path

Disallow

hku www robot,

Rule

Path

Disallow

orbsearch/1.0

Rule

Path

Disallow

packrat/1.0

Rule

Path

Disallow

patric/0.01a

Rule

Path

Disallow

peregrinator-mathematics/0.7

Rule

Path

Disallow

duppies

Rule

Path

Disallow

pioneer

Rule

Path

Disallow

pgp-ka/1.2

Rule

Path

Disallow

resume robot

Rule

Path

Disallow

road runner: imagescape robot (lim@cs.leidenuniv.nl)

Rule

Path

Disallow

robbie/0.1

Rule

Path

Disallow

computingsite robi/1.0 (robi@computingsite.com)

Rule

Path

Disallow

roverbot

Rule

Path

Disallow

safetynet robot 0.1,

Rule

Path

Disallow

not available

Rule

Path

Disallow

senrigan/xxxxxx

Rule

Path

Disallow

sg-scout

Rule

Path

Disallow

shai'hulud

Rule

Path

Disallow

simbot/1.0

Rule

Path

Disallow

open text site crawler v1.0

Rule

Path

Disallow

sitetech-rover

Rule

Path

Disallow

esismartspider/2.0

Rule

Path

Disallow

snooper/b97_01

Rule

Path

Disallow

solbot/1.0 lwp/5.07

Rule

Path

Disallow

spanner/1.0 (linux 2.0.27 i586)

Rule

Path

Disallow

no

Rule

Path

Disallow

mozilla/3.0 (black widow v1.1.0; linux 2.0.27; dec 31 1997 12:25:00

Rule

Path

Disallow

tarantula/1.0

Rule

Path

Disallow

tarspider

Rule

Path

Disallow

dlw3robot/x.y (in tclx by http://hplyot.obspm.fr/~dl/)

Rule

Path

Disallow

templeton/

Rule

Path

Disallow

titin/0.2

Rule

Path

Disallow

titan/0.1

Rule

Path

Disallow

ucsd-crawler

Rule

Path

Disallow

urlck/1.2.3

Rule

Path

Disallow

valkyrie/1.0 libwww-perl/0.40

Rule

Path

Disallow

victoria/1.0

Rule

Path

Disallow

vision-search/3.0'

Rule

Path

Disallow

vwbot_k/4.2

Rule

Path

Disallow

w3index

Rule

Path

Disallow

w3m2/x.xxx

Rule

Path

Disallow

wwwwanderer v3.0

Rule

Path

Disallow

webcopy/

Rule

Path

Disallow

webcrawler/3.0 robot libwww/5.0a

Rule

Path

Disallow

webfetcher/0.8,

Rule

Path

Disallow

weblayers/0.0

Rule

Path

Disallow

weblinker/0.0 libwww-perl/0.1

Rule

Path

Disallow

no

Rule

Path

Disallow

webmoose/0.0.0000

Rule

Path

Disallow

digimarc webreader/1.2

Rule

Path

Disallow

webs@recruit.co.jp

Rule

Path

Disallow

webvac/1.0

Rule

Path

Disallow

webwalk

Rule

Path

Disallow

webwalker/1.10

Rule

Path

Disallow

webwatch

Rule

Path

Disallow

wget/1.4.0

Rule

Path

Disallow

w3mir

Rule

Path

Disallow

wwwc/0.25 (win95)

Rule

Path

Disallow

none

Rule

Path

Disallow

xget/0.7

Rule

Path

Disallow

nederland.zoek

Rule

Path

Disallow

bizbot04 kirk.overleaf.com

Rule

Path

Disallow

happybot (gserver.kw.net)

Rule

Path

Disallow

californiabrownspider

Rule

Path

Disallow

ei*net/0.1 libwww/0.1

Rule

Path

Disallow

ibot/1.0 libwww-perl/0.40

Rule

Path

Disallow

merritt/1.0

Rule

Path

Disallow

statfetcher/1.0

Rule

Path

Disallow

teachersoft/1.0 libwww/2.17

Rule

Path

Disallow

www collector

Rule

Path

Disallow

processor/0.0alpha libwww-perl/0.20

Rule

Path

Disallow

wobot/1.0 from 206.214.202.45

Rule

Path

Disallow

libertech-rover www.libertech.com?

Rule

Path

Disallow

whowhere robot

Rule

Path

Disallow

iti spider

Rule

Path

Disallow

w3index

Rule

Path

Disallow

mycnnspider

Rule

Path

Disallow

summycrawler

Rule

Path

Disallow

ogspider

Rule

Path

Disallow

linklooker

Rule

Path

Disallow

cyberspyder (amant@www.cyberspyder.com)

Rule

Path

Disallow

slowbot

Rule

Path

Disallow

heraspider

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

bizbot003

Rule

Path

Disallow

webwalker

Rule

Path

Disallow

sandbot

Rule

Path

Disallow

enigmabot

Rule

Path

Disallow

spyder3.microsys.com

Rule

Path

Disallow

www.freeloader.com.

Rule

Path

Disallow

metagopher

Rule

Path

Disallow

Warnings

4 invalid lines.

djsilver.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

mediapartners-google*

*

dittospydervscooter

webferret

due to a deficiency in java it's not currently possible to set the user-agent.

aspider/0.09

auresys/1.0

backrub/*.*

blackwidow

cactvs chemistry spider

digimarc cgireader/1.0

checkbot/x.xx lwp/5.x

cmc/0.01

combine/0.0

root/0.1

cs-hkust-indexserver/1.0

emacs-w3/v[0-9\.]+

emailsiphon

emc spider

explorersearch

explorer

extractorpro

felixide/1.0

hazel's ferret web hopper,

esirover v1.0

fido/0.9 harvest/1.4.pl2

hï¿½mï¿½hï¿½kki/0.2

kit-fireball/2.0 libwww/5.0a

fish-search-robot

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

robot du crim 1.0a

freecrawl

funnelweb-1.0

gcreep/1.0

geturl.rexx v1.05

golem/1.1

gromit/1.0

gulliver/1.1

yes

aitcsrobot/1.1

htdig/3.0b3

htmlgobble v2.2

gestalticonoclast/1.0 libwww-fm/2.17

ingrid/0.1

incywincy/1.0b1

informant

infospiders/0.1

inspectorwww/1.0 http://www.greenpac.com/inspectorwww.html

israelisearch/1.0

jcrawler/0.2

jeeves v0.05alpha (perl, lwp, lglb@doc.ic.ac.uk)

jobot/0.1alpha libwww-perl/4.0

joebot,

jubiirobot

jumpstation

katipo/1.0

kdd-explorer/0.1

labelgrab/1.1

linkwalker

logo.gif crawler

magpie/1.0

mediafox/x.y

merzscope

nec-meshexplorer

momspider/1.00 libwww-perl/0.40

monster/vx.x.x -$type ($ostype)

motor/0.2

muscatferret

mwdsearch/0.1

netcarta cyberpilot pro

netmechanic

netscoop/1.0 libwww/5.0a

nhsewalker/3.0

nomad-v2.x

northstar

djsilver.com
robots.txt

dittospyder
vscooter

backrub/.