alanwest.com
robots.txt

Robots Exclusion Standard data for alanwest.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	alanwest.com
Base Domain	alanwest.com
Scan Status	Ok
Last Scan	2024-11-10T22:43:11+00:00
Next Scan	2024-11-24T22:43:11+00:00

Last Scan

Scanned	2024-11-10T22:43:11+00:00
URL	https://alanwest.com/robots.txt
Domain IPs	2001:8b0:dfaa:1af8::32, 81.187.2.252
Response IP	81.187.2.252
Found	Yes
Hash	97fa294030b92b1bc3ba93f976e5e269339dc1946067b12f61e717de5e873061
SimHash	c03dde9e73d3

Groups

mozilla/3.0 (compatible;miner;mailto:miner@miner.com.br)

Rule	Path
Disallow

Rule

Path

Disallow

webferret

Rule	Path
Disallow

Rule

Path

Disallow

due to a deficiency in java it's not currently possible to set the user-agent.

Rule	Path
Disallow

Rule

Path

Disallow

no

Rule	Path
Disallow

Rule

Path

Disallow

arachnophilia

Rule	Path
Disallow

Rule

Path

Disallow

architextspider

Rule	Path
Disallow

Rule

Path

Disallow

aspider/0.09

Rule	Path
Disallow

Rule

Path

Disallow

auresys/1.0

Rule	Path
Disallow

Rule

Path

Disallow

backrub/.

Rule	Path
Disallow

Rule

Path

Disallow

big brother

Rule	Path
Disallow

Rule

Path

Disallow

blackwidow

Rule	Path
Disallow

Rule

Path

Disallow

bspider/1.0 libwww-perl/0.40

Rule	Path
Disallow

Rule

Path

Disallow

cactvs chemistry spider

Rule	Path
Disallow

Rule

Path

Disallow

digimarc cgireader/1.0

Rule	Path
Disallow

Rule

Path

Disallow

checkbot/x.xx lwp/5.x

Rule	Path
Disallow

Rule

Path

Disallow

cmc/0.01

Rule	Path
Disallow

Rule

Path

Disallow

combine/0.0

Rule	Path
Disallow

Rule

Path

Disallow

conceptbot/0.3

Rule	Path
Disallow

Rule

Path

Disallow

crescent internet toolpak http ole control v.1.0

Rule	Path
Disallow

Rule

Path

Disallow

root/0.1

Rule	Path
Disallow

Rule

Path

Disallow

cs-hkust-indexserver/1.0

Rule	Path
Disallow

Rule

Path

Disallow

cyberspyder/2.1

Rule	Path
Disallow

Rule

Path

Disallow

deweb/1.01

Rule	Path
Disallow

Rule

Path

Disallow

dragonbot/1.0 libwww/5.0

Rule	Path
Disallow

Rule

Path

Disallow

eit-link-verifier-robot/0.2

Rule	Path
Disallow

Rule

Path

Disallow

emacs-w3/v[0-9\.]+

Rule	Path
Disallow

Rule

Path

Disallow

emailsiphon

Rule	Path
Disallow

Rule

Path

Disallow

emc spider

Rule	Path
Disallow

Rule

Path

Disallow

explorersearch

Rule	Path
Disallow

Rule

Path

Disallow

explorer

Rule	Path
Disallow

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow

Rule

Path

Disallow

felixide/1.0

Rule	Path
Disallow

Rule

Path

Disallow

hazel's ferret web hopper,

Rule	Path
Disallow

Rule

Path

Disallow

esirover v1.0

Rule	Path
Disallow

Rule

Path

Disallow

fido/0.9 harvest/1.4.pl2

Rule	Path
Disallow

Rule

Path

Disallow

hï¿½ï¿½ï¿½ki/0.2

Rule	Path
Disallow

Rule

Path

Disallow

kit-fireball/2.0 libwww/5.0a

Rule	Path
Disallow

Rule

Path

Disallow

fish-search-robot

Rule	Path
Disallow

Rule

Path

Disallow

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

Rule	Path
Disallow

Rule

Path

Disallow

robot du crim 1.0a

Rule	Path
Disallow

Rule

Path

Disallow

freecrawl

Rule	Path
Disallow

Rule

Path

Disallow

funnelweb-1.0

Rule	Path
Disallow

Rule

Path

Disallow

gcreep/1.0

Rule	Path
Disallow

Rule

Path

Disallow

geturl.rexx v1.05

Rule	Path
Disallow

Rule

Path

Disallow

golem/1.1

Rule	Path
Disallow

Rule

Path

Disallow

gromit/1.0

Rule	Path
Disallow

Rule

Path

Disallow

gulliver/1.1

Rule	Path
Disallow

Rule

Path

Disallow

yes

Rule	Path
Disallow

Rule

Path

Disallow

aitcsrobot/1.1

Rule

Path

Disallow

wired-digital-newsbot/1.5

Rule

Path

Disallow

htdig/3.0b3

Rule

Path

Disallow

htmlgobble v2.2

Rule

Path

Disallow

no

Rule

Path

Disallow

ibm_planetwide,

Rule

Path

Disallow

gestalticonoclast/1.0 libwww-fm/2.17

Rule

Path

Disallow

ingrid/0.1

Rule

Path

Disallow

incywincy/1.0b1

Rule

Path

Disallow

informant

Rule

Path

Disallow

infoseek robot 1.0

Rule

Path

Disallow

infoseek sidewinder

Rule

Path

Disallow

infospiders/0.1

Rule

Path

Disallow

inspectorwww/1.0 http://www.greenpac.com/inspectorwww.html

Rule

Path

Disallow

israelisearch/1.0

Rule

Path

Disallow

jcrawler/0.2

Rule

Path

Disallow

jeeves v0.05alpha (perl, lwp, lglb@doc.ic.ac.uk)

Rule

Path

Disallow

jobot/0.1alpha libwww-perl/4.0

Rule

Path

Disallow

joebot,

Rule

Path

Disallow

jubiirobot

Rule

Path

Disallow

jumpstation

Rule

Path

Disallow

katipo/1.0

Rule

Path

Disallow

kdd-explorer/0.1

Rule

Path

Disallow

ko_yappo_robot/1.0.4(http://yappo.com/info/robot.html)

Rule

Path

Disallow

labelgrab/1.1

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

logo.gif crawler

Rule

Path

Disallow

lycos/x.x

Rule

Path

Disallow

lycos_spider_(t-rex)

Rule

Path

Disallow

magpie/1.0

Rule

Path

Disallow

mediafox/x.y

Rule

Path

Disallow

merzscope

Rule

Path

Disallow

nec-meshexplorer

Rule

Path

Disallow

momspider/1.00 libwww-perl/0.40

Rule

Path

Disallow

monster/vx.x.x -$type ($ostype)

Rule

Path

Disallow

motor/0.2

Rule

Path

Disallow

muscatferret

Rule

Path

Disallow

mwdsearch/0.1

Rule

Path

Disallow

netcarta cyberpilot pro

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

netscoop/1.0 libwww/5.0a

Rule

Path

Disallow

nhsewalker/3.0

Rule

Path

Disallow

nomad-v2.x

Rule

Path

Disallow

northstar

Rule

Path

Disallow

occam/1.0

Rule

Path

Disallow

hku www robot,

Rule

Path

Disallow

orbsearch/1.0

Rule

Path

Disallow

packrat/1.0

Rule

Path

Disallow

patric/0.01a

Rule

Path

Disallow

peregrinator-mathematics/0.7

Rule

Path

Disallow

duppies

Rule

Path

Disallow

pioneer

Rule

Path

Disallow

pgp-ka/1.2

Rule

Path

Disallow

resume robot

Rule

Path

Disallow

road runner: imagescape robot (lim@cs.leidenuniv.nl)

Rule

Path

Disallow

robbie/0.1

Rule

Path

Disallow

computingsite robi/1.0 (robi@computingsite.com)

Rule

Path

Disallow

roverbot

Rule

Path

Disallow

safetynet robot 0.1,

Rule

Path

Disallow

scooter/1.0

Rule

Path

Disallow

not available

Rule

Path

Disallow

senrigan/xxxxxx

Rule

Path

Disallow

sg-scout

Rule

Path

Disallow

shai'hulud

Rule

Path

Disallow

simbot/1.0

Rule

Path

Disallow

open text site crawler v1.0

Rule

Path

Disallow

sitetech-rover

Rule

Path

Disallow

slurp/2.0

Rule

Path

Disallow

esismartspider/2.0

Rule

Path

Disallow

snooper/b97_01

Rule

Path

Disallow

solbot/1.0 lwp/5.07

Rule

Path

Disallow

spanner/1.0 (linux 2.0.27 i586)

Rule

Path

Disallow

no

Rule

Path

Disallow

mozilla/3.0 (black widow v1.1.0; linux 2.0.27; dec 31 1997 12:25:00

Rule

Path

Disallow

tarantula/1.0

Rule

Path

Disallow

tarspider

Rule

Path

Disallow

dlw3robot/x.y (in tclx by http://hplyot.obspm.fr/~dl/)

Rule

Path

Disallow

templeton/

Rule

Path

Disallow

titin/0.2

Rule

Path

Disallow

titan/0.1

Rule

Path

Disallow

ucsd-crawler

Rule

Path

Disallow

urlck/1.2.3

Rule

Path

Disallow

valkyrie/1.0 libwww-perl/0.40

Rule

Path

Disallow

victoria/1.0

Rule

Path

Disallow

vision-search/3.0'

Rule

Path

Disallow

vwbot_k/4.2

Rule

Path

Disallow

w3index

Rule

Path

Disallow

w3m2/x.xxx

Rule

Path

Disallow

wwwwanderer v3.0

Rule

Path

Disallow

webcopy/

Rule

Path

Disallow

webcrawler/3.0 robot libwww/5.0a

Rule

Path

Disallow

webfetcher/0.8,

Rule

Path

Disallow

weblayers/0.0

Rule

Path

Disallow

weblinker/0.0 libwww-perl/0.1

Rule

Path

Disallow

no

Rule

Path

Disallow

webmoose/0.0.0000

Rule

Path

Disallow

digimarc webreader/1.2

Rule

Path

Disallow

webs@recruit.co.jp

Rule

Path

Disallow

webvac/1.0

Rule

Path

Disallow

webwalk

Rule

Path

Disallow

webwalker/1.10

Rule

Path

Disallow

webwatch

Rule

Path

Disallow

wget/1.4.0

Rule

Path

Disallow

w3mir

Rule

Path

Disallow

no

Rule

Path

Disallow

wwwc/0.25 (win95)

Rule

Path

Disallow

none

Rule

Path

Disallow

xget/0.7

Rule

Path

Disallow

nederland.zoek

Rule

Path

Disallow

bizbot04 kirk.overleaf.com

Rule

Path

Disallow

happybot (gserver.kw.net)

Rule

Path

Disallow

californiabrownspider

Rule

Path

Disallow

ei*net/0.1 libwww/0.1

Rule

Path

Disallow

ibot/1.0 libwww-perl/0.40

Rule

Path

Disallow

merritt/1.0

Rule

Path

Disallow

statfetcher/1.0

Rule

Path

Disallow

teachersoft/1.0 libwww/2.17

Rule

Path

Disallow

www collector

Rule

Path

Disallow

processor/0.0alpha libwww-perl/0.20

Rule

Path

Disallow

wobot/1.0 from 206.214.202.45

Rule

Path

Disallow

libertech-rover www.libertech.com?

Rule

Path

Disallow

whowhere robot

Rule

Path

Disallow

iti spider

Rule

Path

Disallow

w3index

Rule

Path

Disallow

mycnnspider

Rule

Path

Disallow

summycrawler

Rule

Path

Disallow

ogspider

Rule

Path

Disallow

linklooker

Rule

Path

Disallow

cyberspyder (amant@www.cyberspyder.com)

Rule

Path

Disallow

slowbot

Rule

Path

Disallow

heraspider

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

bizbot003

Rule

Path

Disallow

webwalker

Rule

Path

Disallow

sandbot

Rule

Path

Disallow

enigmabot

Rule

Path

Disallow

spyder3.microsys.com

Rule

Path

Disallow

www.freeloader.com.

Rule

Path

Disallow

googlebot

Rule

Path

Disallow

metagopher

Rule

Path

Disallow

*

Rule

Path

Disallow

/cgi-bin/

Comments

Robots.txt file from http://www.searchengineworld.com
Built from text file
http://info.webcrawler.com/mak/projects/robots/active/all.txt
This restricts access to only known and registered robots.
/* send all your seo requests tpo abuse@google.com*/

Warnings

6 invalid lines.

alanwest.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

mozilla/3.0 (compatible;miner;mailto:miner@miner.com.br)

webferret

due to a deficiency in java it's not currently possible to set the user-agent.

no

arachnophilia

architextspider

aspider/0.09

auresys/1.0

backrub/*.*

big brother

blackwidow

bspider/1.0 libwww-perl/0.40

cactvs chemistry spider

digimarc cgireader/1.0

checkbot/x.xx lwp/5.x

cmc/0.01

combine/0.0

conceptbot/0.3

crescent internet toolpak http ole control v.1.0

root/0.1

cs-hkust-indexserver/1.0

cyberspyder/2.1

deweb/1.01

dragonbot/1.0 libwww/5.0

eit-link-verifier-robot/0.2

emacs-w3/v[0-9\.]+

emailsiphon

emc spider

explorersearch

explorer

extractorpro

felixide/1.0

hazel's ferret web hopper,

esirover v1.0

fido/0.9 harvest/1.4.pl2

hï¿½ï¿½ï¿½ki/0.2

kit-fireball/2.0 libwww/5.0a

fish-search-robot

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

robot du crim 1.0a

freecrawl

funnelweb-1.0

gcreep/1.0

geturl.rexx v1.05

golem/1.1

gromit/1.0

gulliver/1.1

yes

aitcsrobot/1.1

wired-digital-newsbot/1.5

htdig/3.0b3

htmlgobble v2.2

no

ibm_planetwide,

gestalticonoclast/1.0 libwww-fm/2.17

ingrid/0.1

incywincy/1.0b1

informant

infoseek robot 1.0

infoseek sidewinder

infospiders/0.1

inspectorwww/1.0 http://www.greenpac.com/inspectorwww.html

israelisearch/1.0

jcrawler/0.2

jeeves v0.05alpha (perl, lwp, lglb@doc.ic.ac.uk)

jobot/0.1alpha libwww-perl/4.0

joebot,

jubiirobot

jumpstation

katipo/1.0

kdd-explorer/0.1

ko_yappo_robot/1.0.4(http://yappo.com/info/robot.html)

labelgrab/1.1

linkwalker

logo.gif crawler

alanwest.com
robots.txt

backrub/.