Log In Sign Up

maimonides.org
robots.txt

Robots Exclusion Standard data for maimonides.org

Resource Scan

Scan Details

Site Domain maimonides.org
Base Domain maimonides.org
Scan Status Ok
Last Scan2025-04-26T02:48:48+00:00
Next Scan 2025-05-26T02:48:48+00:00

Last Scan

Scanned2025-04-26T02:48:48+00:00
URL https://maimonides.org/robots.txt
Domain IPs 54.84.7.4
Response IP 54.84.7.4
Found Yes
Hash 7528a62134b82d35a34c03fcf4cc4d061f49668ba2c7fc883f797ac03fde27d6
SimHash 59859b5663de

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

slurp

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Disallow /default.aspx?RelId=60586221

abachobot

Rule Path
Disallow /

about/0.1libwww-perl/5.47

Rule Path
Disallow /

acke.dc.luth.se

Rule Path
Disallow /

acme.spider

Rule Path
Disallow /

acoirobot

Rule Path
Disallow /

acoon robot v1.50.001

Rule Path
Disallow /

ad muncher v4.xx.x

Rule Path
Disallow /

aesop_com_spiderman

Rule Path
Disallow /

ahoy! the homepage finder

Rule Path
Disallow /

aladin/3.324

Rule Path
Disallow /

alkaline

Rule Path
Disallow /

alkalinebot/1.3

Rule Path
Disallow /

alkalinebot/1.4 (1.4.0326.0 rtm)

Rule Path
Disallow /

allesklar/0.1 libwww-perl/5.46

Rule Path
Disallow /

alligator 1.31 (www.nearsoftware.com)

Rule Path
Disallow /

altavista intranet v2.0 avs eval search@freeit.com

Rule Path
Allow /

altavista intranet v2.0 compaq altavista eval sveand@altavista.net

Rule Path
Allow /

altavista intranet v2.0 evreka.com crawler@evreka.com

Rule Path
Allow /

altavista robot

Rule Path
Allow /

altavista robot ??

Rule Path
Allow /

altavista v2.0b crawler@evreka.com

Rule Path
Allow /

amiga voyager browser, amiga

Rule Path
Disallow /

amiga-aweb/3.4.167se

Rule Path
Disallow /

ananzi

Rule Path
Disallow /

annomille spider 0.1 alpha - http://www.annomille.it

Rule Path
Disallow /

anonymized by proxyos: http://www.megaproxy.com

Rule Path
Disallow /

anonymizer/1.1

Rule Path
Disallow /

anthill

Rule Path
Disallow /

antibot-v1.1.5/i586-linux-2.2

Rule Path
Disallow /

aol web tv

Rule Path
Allow /

aol, mac

Rule Path
Allow /

aol, win 3.x

Rule Path
Allow /

a-online search

Rule Path
Disallow /

aport

Rule Path
Disallow /

appie 1.1 (www.walhello.com)

Rule Path
Disallow /

arachnoidea (arachnoidea@euroseek.com)

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

arale

Rule Path
Disallow /

araneo

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

aretha

Rule Path
Disallow /

ariadne

Rule Path
Disallow /

arks

Rule Path
Disallow /

aspider (associative spider)

Rule Path
Disallow /

aspseek/1.2.4a

Rule Path
Disallow /

aspseek/1.2.5

Rule Path
Disallow /

aspseek/1.2.6

Rule Path
Disallow /

aspseek/1.2.8pre

Rule Path
Disallow /

aspseek/1.2.9d

Rule Path
Disallow /

assort/0.10

Rule Path
Disallow /

atn worldwide

Rule Path
Disallow /

atomz.com search robot

Rule Path
Disallow /

atomz/1.0

Rule Path
Disallow /

auresys

Rule Path
Disallow /

av fetch 1.0

Rule Path
Disallow /

avsearch-3.0(altavista/avc)

Rule Path
Allow /

backrub

Rule Path
Disallow /

bannerkiller

Rule Path
Disallow /

big brother

Rule Path
Disallow /

bigbrother/1.6e

Rule Path
Disallow /

bizbot003

Rule Path
Disallow /

bizbot04 kirk.overleaf.com

Rule Path
Disallow /

bjaaland

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

bmclient

Rule Path
Disallow /

borg-bot

Rule Path
Disallow /

bot mailto:craftbot@yahoo.com

Rule Path
Disallow /

bright.net caching robot

Rule Path
Disallow /

bsdseek/1.0

Rule Path
Disallow /

bspider

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bumblebee@relevare.com

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

calif

Rule Path
Disallow /

californiabrownspider

Rule Path
Disallow /

cancer information and support international;

Rule Path
Disallow /

cassandra

Rule Path
Disallow /

ceramic tile installation guide (http://www.floorstransformed.com)

Rule Path
Disallow /

cg-eye interactive

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

checkbot/1.51 lwp/5.18

Rule Path
Disallow /

checkbot/1.6x lwp/5.46

Rule Path
Disallow /

checkurl

Rule Path
Disallow /

checkweb

Rule Path
Disallow /

christcrawler.com

Rule Path
Disallow /

churl

Rule Path
Disallow /

cienciaficcion.net

Rule Path
Disallow /

cmc/0.01

Rule Path
Disallow /

coldfusion

Rule Path
Disallow /

collective

Rule Path
Disallow /

combine system

Rule Path
Disallow /

combine/0.0

Rule Path
Disallow /

commerce browser center

Rule Path
Disallow /

computingsite robi/1.0

Rule Path
Disallow /

conceptbot

Rule Path
Disallow /

contact

Rule Path
Disallow /

contype

Rule Path
Disallow /

coolbot

Rule Path
Disallow /

cosmos/0.8_(robot@xyleme.com)

Rule Path
Disallow /

cosmos/0.9_(robot@xyleme.com)

Rule Path
Disallow /

crawler

Rule Path
Disallow /

crawler admin@crawler.de

Rule Path
Disallow /

crawler v 0.2.x admin@crawler.de

Rule Path
Disallow /

crawler@alexa.com

Rule Path
Disallow /

crawlerboy pinpoint.com

Rule Path
Disallow /

cuam ver0.050bx

Rule Path
Disallow /

curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (openssl 0.9.6)

Rule Path
Disallow /

curl/7.9.3 (win32) libcurl 7.9.3

Rule Path
Disallow /

curryguide sitescan 1.1

Rule Path
Disallow /

cusco

Rule Path
Disallow /

custom spider www.bisnisseek.com /1.0

Rule Path
Disallow /

cybercity.fr user robot / faked user agent ?

Rule Path
Disallow /

cyberdog/2.0 (macintosh; 68k)

Rule Path
Disallow /

cyberspyder (amant@www.cyberspyder.com)

Rule Path
Disallow /

cyberspyder link test

Rule Path
Disallow /

cyberspyder link test/2.1.12 (admin@mspennyworth.com)

Rule Path
Disallow /

da 3.5 (www.lidan.com)

Rule Path
Disallow /

da 4.0

Rule Path
Disallow /

da 4.0 (www.downloadaccelerator.com)

Rule Path
Disallow /

da 5.0

Rule Path
Disallow /

dallas.mt.cs.cmu.edu

Rule Path
Disallow /

darkwing.cadvision.com

Rule Path
Disallow /

daviesbot/1.7 (www.wholeweb.net)

Rule Path
Disallow /

dbdig(http://www.prairielandconsulting.com)

Rule Path
Disallow /

dbrowse 1.4d

Rule Path
Disallow /

dds explorer v1.0 beta

Rule Path
Disallow /

deweb(c) katalog/index

Rule Path
Disallow /

die blinde kuh

Rule Path
Disallow /

dienstspider

Rule Path
Disallow /

diff. ips / services (for example phenominet.com link checking)

Rule Path
Disallow /

diff. ips / services (for example: in conjunction w. weblink's checker)

Rule Path
Disallow /

different ips using the httpclient library (mostly link checking)

Rule Path
Disallow /

digger

Rule Path
Disallow /

digger/1.0 jdk/1.3.0rc3

Rule Path
Disallow /

digimarc marcspider

Rule Path
Disallow /

digimarc marcspider/cgi

Rule Path
Disallow /

digital integrity robot

Rule Path
Disallow /

digout4u

Rule Path
Disallow /

diibot/1.2

Rule Path
Disallow /

direct hit grabber

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dlman

Rule Path
Disallow /

dnabot

Rule Path
Disallow /

docomo/1.0/n210i/c10

Rule Path
Disallow /

download demon/3.x.x.x

Rule Path
Disallow /

download express

Rule Path
Disallow /

download express 1.0

Rule Path
Disallow /

download manager

Rule Path
Disallow /

download wonder

Rule Path
Disallow /

downloadmanager ?

Rule Path
Disallow /

dragonbot

Rule Path
Disallow /

dsl.net user robot

Rule Path
Disallow /

dsurf15a va

Rule Path
Disallow /

dsurf15a xx

Rule Path
Disallow /

dtsearchspider

Rule Path
Disallow /

dual proxy

Rule Path
Disallow /

dwcp (dridus' web cataloging project)

Rule Path
Disallow /

ebiness

Rule Path
Disallow /

ebrowse 1.4b

Rule Path
Disallow /

ecatch/3.0

Rule Path
Disallow /

echo!/2.0

Rule Path
Disallow /

e-collector

Rule Path
Disallow /

ei*net/0.1 libwww/0.1

Rule Path
Disallow /

eit link verifier robot

Rule Path
Disallow /

elfinbot

Rule Path
Disallow /

emacs-w3 search engine

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf 1.00

Rule Path
Disallow /

enigmabot

Rule Path
Disallow /

enterprise_search/1.0

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html)

Rule Path
Disallow /

esismartspider

Rule Path
Disallow /

esther

Rule Path
Disallow /

esurf15a 15

Rule Path
Disallow /

evliya celebi

Rule Path
Disallow /

excalibur internet spider v6.5.4

Rule Path
Disallow /

express webpictures (www.express-soft.com)

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

fake ?

Rule Path
Disallow /

fake entry ??

Rule Path
Disallow /

faked user agent

Rule Path
Disallow /

fastcrawler

Rule Path
Disallow /

fastcrawler 3.0.1 (crawler@1klik.dk)

Rule Path
Disallow /

fast-webcrawler/2.2.10 (multimedia search) (crawler@fast.no; http://www.fast.

Rule Path
Disallow /

fast-webcrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear

Rule Path
Disallow /

fast-webcrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear

Rule Path
Disallow /

fast-webcrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsear

Rule Path
Disallow /

fast-webcrawler/3.2 test

Rule Path
Disallow /

fast-webcrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawl

Rule Path
Disallow /

fast-webcrawler/3.4/nirvana (crawler@fast.no; http://fast.no/support.php?c=fa

Rule Path
Disallow /

fast-webcrawler/3.4/partnersite (crawler@fast.no; http://fast.no/support.php?

Rule Path
Disallow /

fast-webcrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c

Rule Path
Disallow /

favorg

Rule Path
Disallow /

favorites sweeper v.2.03

Rule Path
Disallow /

felix ide

Rule Path
Disallow /

fetchrover

Rule Path
Disallow /

fido

Rule Path
Disallow /

filehound 1.2

Rule Path
Disallow /

firefly/1.0

Rule Path
Disallow /

firefly/1.0 (compatible; mozilla 4.0; msie 5.5)

Rule Path
Disallow /

fish search

Rule Path
Disallow /

flashget

Rule Path
Disallow /

floortransformed.com robot (link ckecking ??)

Rule Path
Disallow /

fluid dynamics search engine robot

Rule Path
Disallow /

flunky

Rule Path
Disallow /

fly/6.01 libwww/4.0d

Rule Path
Disallow /

fouineur

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

fsurf15a 01

Rule Path
Disallow /

funnelweb

Rule Path
Disallow /

gais robot/1.0b2

Rule Path
Disallow /

gaislab data gatherer, gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.c

Rule Path
Disallow /

gammaspider, focusedcrawler

Rule Path
Disallow /

gazz

Rule Path
Disallow /

gazz/2.1 (gazz@nttrd.com)

Rule Path
Disallow /

gcreep

Rule Path
Disallow /

gecko 0.7 linux pentium ii

Rule Path
Disallow /

gecko 0.9 linux pentium ii

Rule Path
Disallow /

geckobot

Rule Path
Disallow /

getbot

Rule Path
Disallow /

getright/3.3.3

Rule Path
Disallow /

getright/4.5xx

Rule Path
Disallow /

getright/4.x

Rule Path
Disallow /

getterroboplus puu

Rule Path
Disallow /

geturl

Rule Path
Disallow /

gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/)

Rule Path
Disallow /

gigabot/1.0

Rule Path
Disallow /

gnodspider (www.gnod.net)

Rule Path
Disallow /

gnu file downloader

Rule Path
Disallow /

go!zilla 3.x (www.gozilla.com)

Rule Path
Disallow /

go!zilla/4.0.0.26

Rule Path
Disallow /

go-ahead-got-it/1.1

Rule Path
Disallow /

golem

Rule Path
Disallow /

goofer/0.2

Rule Path
Disallow /

gotit web accelerator

Rule Path
Disallow /

grapnel/0.01 experiment

Rule Path
Disallow /

griffon

Rule Path
Disallow /

gromit

Rule Path
Disallow /

gseek.com (site is offline) robot

Rule Path
Disallow /

gulliver/1.3

Rule Path
Disallow /

gulper bot

Rule Path
Disallow /

gulper web bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/link/gulperbot)

Rule Path
Disallow /

hambot

Rule Path
Disallow /

happybot (gserver.kw.net)

Rule Path
Disallow /

harvest

Rule Path
Disallow /

harvest-ng/1.0.2

Rule Path
Disallow /

havindex

Rule Path
Disallow /

heraspider

Rule Path
Disallow /

hi (html index) search

Rule Path
Disallow /

hippias/0.9 beta

Rule Path
Disallow /

hku www octopus

Rule Path
Disallow /

hloader

Rule Path
Disallow /

home.com user agent

Rule Path
Disallow /

homepagesearch(hpsearch.uni-trier.de)

Rule Path
Disallow /

hometown spider pro

Rule Path
Disallow /

hotjava browser plus html component 1.1.x

Rule Path
Disallow /

hotjava/1.0.1/jre1.1.x

Rule Path
Disallow /

ht://dig

Rule Path
Disallow /

html link validator (www.lithopssoft.com)

Rule Path
Disallow /

html_analyzer

Rule Path
Disallow /

htmlgobble

Rule Path
Disallow /

http://@nonymouse.com/ (linux)

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

hyper-decontextualizer

Rule Path
Disallow /

hsmshskki

Rule Path
Disallow /

i, robot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver/1.6

Rule Path
Disallow /

iajabot

Rule Path
Disallow /

ibm_planetwide

Rule Path
Disallow /

ibot/1.0 libwww-perl/0.40

Rule Path
Disallow /

ibrowse/2.2 (amigaos 3.5)

Rule Path
Disallow /

ibrowse/2.2 (windows 3.1)

Rule Path
Disallow /

icab/2.5.2 (macintosh; i; ppc)

Rule Path
Disallow /

ideare - signsite/1.x

Rule Path
Disallow /

ie 2.x, mac power pc

Rule Path
Disallow /

ie 2.x, winnt

Rule Path
Disallow /

ie 3.x, aol, win95

Rule Path
Disallow /

ie 3.x, win 3.1

Rule Path
Disallow /

ie 3.x, win95

Rule Path
Disallow /

ie 3.x, winnt

Rule Path
Disallow /

ie 3.x, winxp

Rule Path
Disallow /

ie 4.x, aol, win98

Rule Path
Disallow /

ie 4.x, mac power pc

Rule Path
Disallow /

ie 4.x, win95

Rule Path
Disallow /

ie 4.x, winnt

Rule Path
Disallow /

ie 5 fake user agent

Rule Path
Disallow /

ie 5.5 compatible browser

Rule Path
Disallow /

ie 5.5, win2000 / jaring.my user agent

Rule Path
Disallow /

ie 5.5, win2000 / user agent

Rule Path
Disallow /

ie 5.5, win2000 with ms .net sdk

Rule Path
Disallow /

ie 5.x, mac powerpc, athome user

Rule Path
Disallow /

ie pda browser, windows ce

Rule Path
Disallow /

iefav172free

Rule Path
Disallow /

image.kapsi.net

Rule Path
Disallow /

imagelock

Rule Path
Disallow /

incywincy

Rule Path
Disallow /

incywincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprove

Rule Path
Disallow /

incywincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovem

Rule Path
Disallow /

incywincy(http://www.look.com)

Rule Path
Disallow /

incywincy(http://www.loopimprovements.com/robot.html)

Rule Path
Disallow /

incywincy/2.1(loopimprovements.com/robot.html)

Rule Path
Disallow /

inet library

Rule Path
Disallow /

infonavirobot(f107)

Rule Path
Disallow /

informant

Rule Path
Disallow /

infoseek robot 1.0

Rule Path
Disallow /

infoseek sidewinder

Rule Path
Disallow /

infoseek sidewinder/0.9

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

ingrid

Rule Path
Disallow /

inktomi search

Rule Path
Disallow /

inktomi slurp

Rule Path
Disallow /

inspector web

Rule Path
Disallow /

installshield digitalwizard

Rule Path
Disallow /

intelliagent

Rule Path
Disallow /

internet cruiser robot

Rule Path
Disallow /

internet direct library for borland (used as e-mail collector)

Rule Path
Disallow /

internet explorer 4.0 url check

Rule Path
Disallow /

internet ninja 6.0

Rule Path
Disallow /

internet shinchakubin

Rule Path
Disallow /

internetlinkagent/3.1

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

iplexx austria spider ??

Rule Path
Disallow /

irix 6.5

Rule Path
Disallow /

iron33

Rule Path
Disallow /

israeli-search

Rule Path
Disallow /

iti spider

Rule Path
Disallow /

jack

Rule Path
Disallow /

janas (ideare.com / tiscali.it) robot

Rule Path
Disallow /

java 1.1

Rule Path
Disallow /

java development kit

Rule Path
Disallow /

java1.0.21.0

Rule Path
Disallow /

java1.1.xx.x

Rule Path
Disallow /

java1.3.0rc1

Rule Path
Disallow /

java1.3.x

Rule Path
Disallow /

java1.4.0

Rule Path
Disallow /

javabee

Rule Path
Disallow /

jbot java web robot

Rule Path
Disallow /

jchecklinks java hyperlink validator

Rule Path
Disallow /

jchecklinks/0.1 rpt-httpclient/0.3-1

Rule Path
Disallow /

jcrawler

Rule Path
Disallow /

jdk/1.1

Rule Path
Disallow /

jeeves

Rule Path
Allow /

jeode/1.7.2

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jigsaw/2.2.0 w3c_css_validator_jfouffa/2.0

Rule Path
Disallow /

jobo java web robot

Rule Path
Disallow /

jobo/@jobo_version@(http://www.matuschek.net/jobo.html)

Rule Path
Disallow /

jobot

Rule Path
Disallow /

joebot

Rule Path
Disallow /

jumpstation

Rule Path
Disallow /

katipo

Rule Path
Disallow /

kdd-explorer

Rule Path
Disallow /

ke_1.0/2.0 libwww/5.2.8

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

kilroy

Rule Path
Disallow /

kit-fireball

Rule Path
Disallow /

kit-fireball/2.0

Rule Path
Disallow /

kit-fireball/2.0 (compatible; mozilla 4.0; msie 5.5)

Rule Path
Disallow /

k-meleon/0.6 (windows; u; windows nt 5.1; en-us; rv:0.9.5) gecko/20011011

Rule Path
Disallow /

ko_yappo_robot

Rule Path
Disallow /

konqueror 2.0.x, x11

Rule Path
Disallow /

konqueror 2.1.x, x11

Rule Path
Disallow /

konqueror 2.2.x

Rule Path
Disallow /

konqueror 2.2.x, linux

Rule Path
Disallow /

kulturarw3/0.1

Rule Path
Disallow /

labelgrabber

Rule Path
Disallow /

larbin

Rule Path
Disallow /

larbin (samualt9@bigfoot.com)

Rule Path
Disallow /

larbin_2.1.1 larbin2.1.1@somewhere.com

Rule Path
Disallow /

larbin_2.2.0 (crawl@compete.com)

Rule Path
Disallow /

larbin_2.2.0 crawl@compete.com

Rule Path
Disallow /

larbin_2.2.1_de_viennot (laurent.viennot@inria.fr)

Rule Path
Disallow /

larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp)

Rule Path
Disallow /

larbin_2.2.2 sugayama@lab7.kuis.kyoto-u.ac.jp

Rule Path
Disallow /

larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr)

Rule Path
Disallow /

larbin_2.2.2_guillaume guillaume@liafa.jussieu.fr

Rule Path
Disallow /

larbin_2.6.0 (larbin2.6.0@unspecified.mail)

Rule Path
Disallow /

larbin_2.6.1 (larbin2.6.1@unspecified.mail)

Rule Path
Disallow /

larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)

Rule Path
Disallow /

larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/)

Rule Path
Disallow /

lecodechecker/3.0 libgetdoc/1.0

Rule Path
Disallow /

legs

Rule Path
Disallow /

leia/2.90

Rule Path
Disallow /

leia/3.01pr (leiacrawler; [snip])

Rule Path
Disallow /

lexibot/1.00

Rule Path
Disallow /

libertech-rover www.libertech.com?

Rule Path
Disallow /

libweb/clshttp -- hiongun@kt.co.kr

Rule Path
Disallow /

libwww-perl/5.41

Rule Path
Disallow /

libwww-perl/5.48

Rule Path
Disallow /

libwww-perl/5.50

Rule Path
Disallow /

libwww-perl/5.52 fp/2.1

Rule Path
Disallow /

libwww-perl/5.52 fp/4.0

Rule Path
Disallow /

libwww-perl/5.53

Rule Path
Disallow /

libwww-perl/5.63

Rule Path
Disallow /

link checking software ?

Rule Path
Disallow /

link valet online 1.1

Rule Path
Disallow /

link validator

Rule Path
Disallow /

linkalarm link validation

Rule Path
Disallow /

linkalarm/2.x

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

linkguard.com link validation (service is offline)

Rule Path
Disallow /

linklooker

Rule Path
Disallow /

linkprover 2.1

Rule Path
Disallow /

links - http://gossamer-threads.com/scripts/links/

Rule Path
Disallow /

links (0.97pre12; linux 2.2.14-5.0 i686; 80x24)

Rule Path
Disallow /

links 2.0 (http://gossamer-threads.com/scripts/links/)

Rule Path
Disallow /

links sql (http://gossamer-threads.com/scripts/links-sql/)

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkscan/8.x unix

Rule Path
Disallow /

linkscan/9.0 unix

Rule Path
Disallow /

linkscan/9.0g unix

Rule Path
Disallow /

linksmanager.com (http://linksmanager.com/linkchecker.html)

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

lockon

Rule Path
Disallow /

lockstep spider/1.0

Rule Path
Disallow /

logo.gif crawler

Rule Path
Disallow /

lotus notes 5.0 browser

Rule Path
Disallow /

lotus notes browser

Rule Path
Disallow /

lotus-notes/4.5 ( windows-nt )

Rule Path
Disallow /

lwp::simple/5.22

Rule Path
Disallow /

lwp::simple/5.36

Rule Path
Disallow /

lwp::simple/5.50

Rule Path
Disallow /

lwp::simple/5.53

Rule Path
Disallow /

lwp::simple/5.63

Rule Path
Disallow /

lwp-trivial/1.32

Rule Path
Disallow /

lwp-trivial/1.34

Rule Path
Disallow /

lwp-trivial/1.35

Rule Path
Disallow /

lycos

Rule Path
Disallow /

lycos_spider_(modspider)

Rule Path
Disallow /

lycos_spider_(t-rex)

Rule Path
Disallow /

lynx 2.6 browser

Rule Path
Disallow /

lynx 2.8 browser

Rule Path
Disallow /

lynx/2.6 libwww-fm/2.14

Rule Path
Disallow /

lynx/2.8.3dev.9 libwww-fm/2.14 ssl-mm/1.4.1 openssl/0.9.6

Rule Path
Disallow /

mac wwwworm

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

magpie

Rule Path
Disallow /

mantraagent

Rule Path
Disallow /

mariner/5.1b [de] (win95; i ;kolibri gncwebbot)

Rule Path
Disallow /

marvin v0.3

Rule Path
Disallow /

marvin/infoseek

Rule Path
Disallow /

mass downloader 2.2

Rule Path
Disallow /

mata hari/2.00

Rule Path
Disallow /

mattie

Rule Path
Disallow /

mediacrawler-1.0 (experimental)

Rule Path
Disallow /

mediafox

Rule Path
Disallow /

mediasearch/0.1

Rule Path
Disallow /

megasheep v1.0 (www.searchuk.com internet sheep)

Rule Path
Disallow /

mercator-1.x

Rule Path
Disallow /

mercator-2.0

Rule Path
Disallow /

mercator-scrub-1.1

Rule Path
Disallow /

merritt/1.0

Rule Path
Disallow /

merzscope

Rule Path
Disallow /

metager-linkchecker

Rule Path
Disallow /

metaproducts download express/1.x

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

microsoft url control - 6.00.8xxx

Rule Path
Disallow /

microsoft.com user agent

Rule Path
Disallow /

miixpc/4.2

Rule Path
Disallow /

mindcrawler

Rule Path
Disallow /

mindspring.com user robot

Rule Path
Disallow /

mister pix version.dll

Rule Path
Disallow /

mnogosearch search engine software

Rule Path
Disallow /

mobile phone browser

Rule Path
Disallow /

moget

Rule Path
Disallow /

moget/2.1 (moget@goo.ne.jp)

Rule Path
Disallow /

momspider

Rule Path
Disallow /

monster

Rule Path
Disallow /

motor

Rule Path
Disallow /

mozi!

Rule Path
Disallow /

mozilla

Rule Path
Disallow /

mozilla (mozilla@somewhere.com)

Rule Path
Disallow /

mozilla/1.1 (compatible; mspie 2.0; windows ce)

Rule Path
Disallow /

mozilla/1.22 (compatible; msie 2.0d; windows nt)

Rule Path
Disallow /

mozilla/2.0 (compatible; aol 3.0; mac_powerpc)

Rule Path
Disallow /

mozilla/2.0 (compatible; aol-iweng 3.0; win16)

Rule Path
Disallow /

mozilla/2.0 (compatible; ask jeeves)

Rule Path
Allow /

mozilla/2.0 (compatible; ezresult -- internet search engine)

Rule Path
Disallow /

mozilla/2.0 (compatible; ms frontpage 3.0)

Rule Path
Disallow /

mozilla/2.0 (compatible; ms frontpage 4.0)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 2.1; mac_powerpc)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 3.0; ak; windows 95)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 3.0; windows 3.1)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 3.02; update a; ak; windows nt)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 3.02; update a; aol 3.0; windows 95)

Rule Path
Disallow /

mozilla/2.0 (compatible; msie 3.0b; win32)

Rule Path
Disallow /

mozilla/2.0 (compatible; newt activex; win32)

Rule Path
Disallow /

mozilla/2.0 (compatible; t-h-u-n-d-e-r-s-t-o-n-e)

Rule Path
Disallow /

mozilla/2.0 compatible; check&get 1.1x (windows 98)

Rule Path
Disallow /

mozilla/2.01 (win16; i)

Rule Path
Disallow /

mozilla/2.02gold (win95; i)

Rule Path
Disallow /

mozilla/3.0 (compatible)

Rule Path
Disallow /

mozilla/3.0 (compatible; fluffy the spider; http://www.searchhippo.com/; info

Rule Path
Disallow /

mozilla/3.0 (compatible; indy library)

Rule Path
Disallow /

mozilla/3.0 (compatible; muscatferret/1.5.4; claude@euroferret.com)

Rule Path
Disallow /

mozilla/3.0 (compatible; muscatferret/1.5; olly@muscat.co.uk)

Rule Path
Disallow /

mozilla/3.0 (compatible; muscatferret/1.6.x; claude@euroferret.com)

Rule Path
Disallow /

mozilla/3.0 (compatible; netpositive/2.2)

Rule Path
Disallow /

mozilla/3.0 (compatible; opera/3.0; windows 3.1) v3.1

Rule Path
Disallow /

mozilla/3.0 (compatible; opera/3.0; windows 95/nt4) 3.2

Rule Path
Disallow /

mozilla/3.0 (compatible; rel software, web link validator 2.x)

Rule Path
Disallow /

mozilla/3.0 (compatible; web link validator 2.x)

Rule Path
Disallow /

mozilla/3.0 (compatible; webcapture 2.0; auto; windows)

Rule Path
Disallow /

mozilla/3.0 (compatible; webinator-dev01.home.iprospect.com/2.56)

Rule Path
Disallow /

mozilla/3.0 (compatible;viking/1.8)

Rule Path
Disallow /

mozilla/3.0 (liberate dtv 1.1)

Rule Path
Disallow /

mozilla/3.0 (planetweb/2.100 js ssl us; dreamcast us)

Rule Path
Disallow /

mozilla/3.0 (slurp.so/goo; slurp@inktomi.com; http://www.inktomi.com/slurp.ht

Rule Path
Disallow /

mozilla/3.0 (slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

mozilla/3.0 (slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

mozilla/3.0 (vagabondo/1.x mt; webagent@wise-guys.nl; http://webagent.wise-gu

Rule Path
Disallow /

mozilla/3.0 (win16; i)

Rule Path
Disallow /

mozilla/3.0 (win95; i)

Rule Path
Disallow /

mozilla/3.0 (winnt; i)

Rule Path
Disallow /

mozilla/3.0 (worldgate gazelle 3.5.1 build 11; freebsd2.2.8-stable)

Rule Path
Disallow /

mozilla/3.0 (x11; i; osf1 v4.0 alpha)

Rule Path
Disallow /

mozilla/3.0 navio_aoltv (11; 13; philips; ph200; 1; r2.0c36_aol.0110optik; r2

Rule Path
Disallow /

mozilla/3.0 webtv/1.2 (compatible; msie 2.0)

Rule Path
Disallow /

mozilla/3.01 (compatible; amigavoyager/2.95; amigaos/mc680x0)

Rule Path
Disallow /

mozilla/3.01 (compatible; links2go similarity engine)

Rule Path
Disallow /

mozilla/3.01 (compatible; netbox/3.5 r92; linux 2.2)

Rule Path
Disallow /

mozilla/3.01-c-macos8 (macintosh; i; ppc)

Rule Path
Disallow /

mozilla/3.01gold (x11; i; linux 2.0.32 i486)

Rule Path
Disallow /

mozilla/3.01gold (x11; i; sunos 5.5.1 sun4m)

Rule Path
Disallow /

mozilla/3.01sgoldc-sgi (x11; i; irix 6.3 ip32)

Rule Path
Disallow /

mozilla/3.04 (compatible; antfresco/2.13; risc os 4.02)

Rule Path
Disallow /

mozilla/3.04 (compatible; ncbrowser/2.35; antfresco/2.17; risc os-nc 5.13 laz

Rule Path
Disallow /

mozilla/3.x (i-opener 1.1; netpliance)

Rule Path
Disallow /

mozilla/4.0 (compatible; advanced email extractor v2.xx)

Rule Path
Disallow /

mozilla/4.0 (compatible; bordermanager 3.0)

Rule Path
Disallow /

mozilla/4.0 (compatible; dnloadmage 1.0)

Rule Path
Disallow /

mozilla/4.0 (compatible; fastcrawler3, support-fastcrawler3@fast.no)

Rule Path
Disallow /

mozilla/4.0 (compatible; fdse robot)

Rule Path
Disallow /

mozilla/4.0 (compatible; grub-client-0.2.x; crawl your stuff with http://grub

Rule Path
Disallow /

mozilla/4.0 (compatible; grub-client-0.3.x; crawl your own stuff with http://

Rule Path
Disallow /

mozilla/4.0 (compatible; ics 1.2.xxx)

Rule Path
Disallow /

mozilla/4.0 (compatible; iplexx spider/1.0, http://www.iplexx.at)

Rule Path
Disallow /

mozilla/4.0 (compatible; lotus-notes/5.0; windows-nt)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows nt; site server 3.0 robot) indones

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.0; windows nt; site server 3.0 robot) webques

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.0; windows nt; snap.home.1)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; aol 4.0; windows 98)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; mac_powerpc)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; msiecrawler; windows 95)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows 95)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows 95; snap.home.5)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows ce; msn companion 2.0; 800x600; c

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows nt windows ce)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows nt)

Rule Path
Allow /

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot) microsof

Rule Path
Allow /

mozilla/4.0 (compatible; msie 5.0; aol 5.0; windows 95; digext; gateway2000;

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; mac_powerpc; athome021)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; win32) via proxy gateway cern-httpd/3.0 li

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows 95) truerobot; 1.5

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows 95) voilabot; 1.6

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows 98; digext; kitv4.7 wanadoo)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; windows me) opera 5.11 [en]

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.0; www.galaxy.com; www.psychedelix.com)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.01; windows 95) via avirt gateway server

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.01; windows 98; madasafish)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.01; windows nt 5.0) (samualt9@bigfoot.com)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.01; windows nt 5.0; netcaptor 6.5.0rc1)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; aol 4.0; windows 98; gobeez (www.gobeez.co

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 95) via proxy gateway cern-httpd/3

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 95; transmission segment; hotbar 2

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 98; kitv4.7 wanadoo)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 98; safexplorer tl)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 98; sympa; katiesoft 7; simulbrows

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 98; win 9x 4.90; btinternet v8.1)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows 98; win 9x 4.90; msiecrawler)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 4.0; obot)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 4.0; qxw03018)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0) active cache request

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0) fetch api request

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0; .net clr 1.0.3705)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0; asptear 1.5)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0; n_o_k_i_a)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; win32) webwasher 3.0

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; windows 98) rel software, web link validat

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; windows 98) web link validator 2.x)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; windows 98; net m@nager v3.02 - www.vinn.c

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; windows nt 5.1; hotbar 3.0)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 6.0; windows nt 5.1; q312461; iopener release 1

Rule Path
Disallow /

mozilla/4.0 (compatible; netcraft web server survey)

Rule Path
Disallow /

mozilla/4.0 (compatible; opera/3.0; windows 4.10) 3.51 [en]

Rule Path
Disallow /

mozilla/4.0 (compatible; powermarks/3.5; windows 95/98/2000/nt)

Rule Path
Disallow /

mozilla/4.0 (compatible; sitekiosk 4.0; msie 5.0; windows 98; sitecoach 1.0)

Rule Path
Disallow /

mozilla/4.0 (compatible; win32; winhttp.winhttprequest.5)

Rule Path
Disallow /

mozilla/4.0 (compatible; www.galaxy.com)

Rule Path
Disallow /

mozilla/4.0 (compatible; www.linkguard.com linkguard online 1.0; windows nt)

Rule Path
Disallow /

mozilla/4.0 (fantombrowser)

Rule Path
Disallow /

mozilla/4.0 (fantomcrew browser)

Rule Path
Disallow /

mozilla/4.0 (hhjhj@yahoo.com)

Rule Path
Disallow /

mozilla/4.0 (sleek spider/1.2)

Rule Path
Disallow /

mozilla/4.0 compatible zyborg/1.0 (zyborg@wisenutbot.com; http://www.wisenutb

Rule Path
Disallow /

mozilla/4.0 compatible zyborg/1.0 for homepage (zyborg@wisenutbot.com; http:/

Rule Path
Disallow /

mozilla/4.0 webtv/2.6 (compatible; msie 4.0)

Rule Path
Disallow /

mozilla/4.0_(compatible;_msie_5.0;_windows_95)_truerobot/1.4 libwww/5.2.8

Rule Path
Disallow /

mozilla/4.0_(compatible;_msie_5.0;_windows_95)_voilabot/1.6 libwww/5.3.2

Rule Path
Disallow /

mozilla/4.01 [en](win95;i)

Rule Path
Disallow /

mozilla/4.02 [en] (x11; i; sunos 5.6 sun4u)

Rule Path
Disallow /

mozilla/4.04 [en] (x11; i; hp-ux b.10.20 9000/712)

Rule Path
Disallow /

mozilla/4.04 [en] (x11; i; irix 5.3 ip22)

Rule Path
Disallow /

mozilla/4.05 (macintosh; i; 68k, nav)

Rule Path
Disallow /

mozilla/4.05 (macintosh; i; ppc, nav)

Rule Path
Disallow /

mozilla/4.05 [en] (x11; i; sunos 4.1.4 sun4m)

Rule Path
Disallow /

mozilla/4.5 (compatible; httrack 3.0x; windows 98)

Rule Path
Disallow /

mozilla/4.5 (compatible; icab 2.5.3; macintosh; i; ppc)

Rule Path
Disallow /

mozilla/4.5 (compatible; omniweb/4.0.5; mac_powerpc)

Rule Path
Disallow /

mozilla/4.5 (compatible; omniweb/4.1-beta-1; mac_powerpc)

Rule Path
Disallow /

mozilla/4.5 [en]c-cck-mcd {ruralnet} (win98; i)

Rule Path
Disallow /

mozilla/4.5 rpt-httpclient/0.3-2

Rule Path
Disallow /

mozilla/4.5b1 [en] (x11; i; linux 2.0.35 i586)

Rule Path
Disallow /

mozilla/4.61 [de] (os/2; i)

Rule Path
Disallow /

mozilla/4.7 (compatible; http://eidetica.com/spider)

Rule Path
Disallow /

mozilla/4.7 (compatible; offbyone; windows 98) webster pro v3.2

Rule Path
Disallow /

mozilla/4.7 (compatible; whizbang)

Rule Path
Disallow /

mozilla/4.7 (compatible; whizbang; http://www.whizbang.com/crawler)

Rule Path
Disallow /

mozilla/4.7 [en]c-cck-mcd {bresnanlink} (win95; u)

Rule Path
Disallow /

mozilla/4.7 [en]c-cck-mcd {yahoo;yip052400} (win95; i)

Rule Path
Allow /

mozilla/4.72 [en] (bacs http://www.ba.be)

Rule Path
Disallow /

mozilla/4.72c-cck-mcd caldera systems openlinux [en] (x11; u; linux 2.2.14 i6

Rule Path
Disallow /

mozilla/4.75c-ja [ja] (x11; u; osf1 v5.1 alpha)

Rule Path
Disallow /

mozilla/4.76 (windows 98; u) opera 5.12 [en]

Rule Path
Disallow /

mozilla/4.76 [en] (x11; u; freebsd 4.4-stable i386)

Rule Path
Disallow /

mozilla/4.76 [en] (x11; u; sunos 5.7 sun4u)

Rule Path
Disallow /

mozilla/4.77c-sgi [en] (x11; u; irix 6.5 ip32)

Rule Path
Disallow /

mozilla/5.0

Rule Path
Disallow /

mozilla/5.0 (compatible; konqueror/2.0.1; x11); supports md5-digest; supports

Rule Path
Disallow /

mozilla/5.0 (compatible; konqueror/2.1.1; x11)

Rule Path
Disallow /

mozilla/5.0 (compatible; konqueror/2.2.2)

Rule Path
Disallow /

mozilla/5.0 (compatible; konqueror/2.2.2; linux 2.4.14-xfs; x11; i686)

Rule Path
Disallow /

mozilla/5.0 (macintosh; u; ppc; en-us; rv:0.9.2) gecko/20010726 netscape6/6.1

Rule Path
Disallow /

mozilla/5.0 (slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

mozilla/5.0 (sunos 5.8 sun4u; u) opera 5.0 [en]

Rule Path
Disallow /

mozilla/5.0 (windows; u; win98; en-us; rv:0.9.2) gecko/20010726 netscape6/6.1

Rule Path
Disallow /

mozilla/5.0 (x11; u; linux 2.4.2-2 i586; en-us; m18) gecko/20010131 netscape6

Rule Path
Disallow /

mozilla/5.0 (x11; u; linux 2.4.2-2 i686; en-us; 0.7) gecko/20010316

Rule Path
Disallow /

mozilla/5.0 (x11; u; linux i686; en-us; rv:0.9.3) gecko/20010801

Rule Path
Disallow /

mozilla/5.0 galeon/1.0.2 (x11; linux i686; u;) gecko/20011224

Rule Path
Disallow /

mozilla/5.0 url-spider

Rule Path
Disallow /

mozilla/5.0 usww.com-spider-for-w8.net

Rule Path
Disallow /

mozilla/6.01 (compatible; noydb v0.42)

Rule Path
Disallow /

ms frontpage 4.x

Rule Path
Disallow /

msfrontpage/4.0

Rule Path
Disallow /

msie 4.0 (win95)

Rule Path
Disallow /

msproxy/2.0

Rule Path
Disallow /

multiblocker browser

Rule Path
Disallow /

multitext/0.1

Rule Path
Disallow /

muncher

Rule Path
Disallow /

murph.cais.net (rapid fire... sigh)

Rule Path
Disallow /

muscat ferret

Rule Path
Disallow /

mwd.search

Rule Path
Disallow /

mycnnspider

Rule Path
Disallow /

mydaemon

Rule Path
Disallow /

mygetright/1.0.0

Rule Path
Disallow /

mygetright/1.0b

Rule Path
Disallow /

nabot/5.0

Rule Path
Disallow /

nabot_1.0

Rule Path
Disallow /

nationaldirectoryaddurl/1.0

Rule Path
Disallow /

nationaldirectory-webspider/1.3

Rule Path
Disallow /

ncbrowser, risc os

Rule Path
Disallow /

ncsa beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)

Rule Path
Disallow /

nec research agent -- compuman at research.nj.nec.com

Rule Path
Disallow /

nec-hayek/1.0

Rule Path
Disallow /

nec-meshexplorer

Rule Path
Disallow /

nederland.zoek

Rule Path
Disallow /

netants/1.2x

Rule Path
Disallow /

netcarta webmap engine

Rule Path
Disallow /

netcomplete/x.xx

Rule Path
Disallow /

netlookout/2.24

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netmechanic vx.0

Rule Path
Disallow /

netpositive beos browser

Rule Path
Disallow /

netprospector javacrawler

Rule Path
Disallow /

netresearchserver/2.2(loopimprovements.com/robot.html)

Rule Path
Disallow /

netscape 2.x gold, win95

Rule Path
Disallow /

netscape 2.x, win3.x, international

Rule Path
Disallow /

netscape 3.x, freebsd

Rule Path
Disallow /

netscape 3.x, irix

Rule Path
Disallow /

netscape 3.x, linux

Rule Path
Disallow /

netscape 3.x, mac

Rule Path
Disallow /

netscape 3.x, osf1 v4.0 alpha

Rule Path
Disallow /

netscape 3.x, risc os

Rule Path
Disallow /

netscape 3.x, sunos

Rule Path
Disallow /

netscape 3.x, win3.x

Rule Path
Disallow /

netscape 3.x, win95

Rule Path
Disallow /

netscape 3.x, winnt

Rule Path
Disallow /

netscape 4.7x japan osf1 alpha

Rule Path
Disallow /

netscape 4.7x, freebsd

Rule Path
Disallow /

netscape 4.7x, sunos

Rule Path
Disallow /

netscape 4.x hp-unix

Rule Path
Disallow /

netscape 4.x irix

Rule Path
Disallow /

netscape 4.x macintosh 68k

Rule Path
Disallow /

netscape 4.x macintosh powerpc

Rule Path
Disallow /

netscape 4.x sunos 4.1.4

Rule Path
Disallow /

netscape 4.x sunos 5.6

Rule Path
Disallow /

netscape 4.x, linux

Rule Path
Disallow /

netscape 4.x, os/2

Rule Path
Disallow /

netscape 6.x, linux, pentium i

Rule Path
Disallow /

netscape 6.x, mac powerpc

Rule Path
Disallow /

netscape 6.x, win98

Rule Path
Disallow /

netscoop

Rule Path
Disallow /

netscpape 4.7x, caldera open linux, pentium iii

Rule Path
Disallow /

netzippy

Rule Path
Disallow /

neuralbot/0.2

Rule Path
Disallow /

newscan-online

Rule Path
Disallow /

nhse web forager

Rule Path
Disallow /

nico/1.0

Rule Path
Disallow /

noago spider

Rule Path
Disallow /

nocilla/1.0

Rule Path
Disallow /

nokia.com network

Rule Path
Disallow /

nokia7110/1.0 (05.01) (google wap proxy/1.0)

Rule Path
Allow /

nokia-waptoolkit/1.2 googlebot(at)googlebot.com

Rule Path
Allow /

nomad

Rule Path
Disallow /

northern light gulliver

Rule Path
Disallow /

novell border manager security suite

Rule Path
Disallow /

nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp)

Rule Path
Disallow /

nttrd.com / infobee.ne.jp robot

Rule Path
Disallow /

nzexplorer

Rule Path
Disallow /

obot ((compatible;win32))

Rule Path
Disallow /

occam

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline explorer 1.*

Rule Path
Disallow /

ogspider

Rule Path
Disallow /

oliverperry

Rule Path
Disallow /

omniweb 4.0.x mac browser

Rule Path
Disallow /

omniweb 4.1.x mac browser

Rule Path
Disallow /

open text index robot

Rule Path
Disallow /

openfind data gatherer

Rule Path
Disallow /

openfind data gatherer, openbot/3.0+(robot-response@openfind.com.tw;+http://w

Rule Path
Disallow /

openfind robot/1.1a2

Rule Path
Disallow /

opentextsitecrawler/2.9.2

Rule Path
Disallow /

opera 3.x winnt

Rule Path
Disallow /

opera 3.x, win3.x

Rule Path
Disallow /

opera 3.x, win95/nt

Rule Path
Disallow /

opera 5.0 linux

Rule Path
Disallow /

opera 5.11 faked winme referer

Rule Path
Disallow /

opera 5.12 win98

Rule Path
Disallow /

opera 5.x, sunos

Rule Path
Disallow /

opera 5.x, win 98

Rule Path
Disallow /

opera 6.0 winnt

Rule Path
Disallow /

opera 6.x linux

Rule Path
Disallow /

opera/5.0 (linux 2.0.38 i386; u) [en]

Rule Path
Disallow /

opera/5.11 (windows me; u) [ru]

Rule Path
Disallow /

opera/5.12 (windows 98; u) [en]

Rule Path
Disallow /

opera/6.0 (linux 2.4.8-26mdk i686; u) [en]

Rule Path
Disallow /

opera/6.0 (windows nt 4.0; u) [de]

Rule Path
Disallow /

oracle imtcrawler

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

orb search

Rule Path
Disallow /

pack rat

Rule Path
Disallow /

packard bell net user robot (*not* google)

Rule Path
Disallow /

pageboy

Rule Path
Disallow /

pagmiedownload

Rule Path
Disallow /

parallelcontextfocuscrawler1.1parallelcontextfocuscrawler1.1

Rule Path
Disallow /

parasite

Rule Path
Disallow /

parasite/1.0b (http://www.ianett.com/parasite/)

Rule Path
Disallow /

patric

Rule Path
Disallow /

pavuk/0.9pl29b i686-pc-linux-gnu

Rule Path
Disallow /

pbrowse 1.4b

Rule Path
Disallow /

pda pocket ie 2.x, windows ce

Rule Path
Disallow /

pegasus

Rule Path
Disallow /

perlcrawler 1.0

Rule Path
Disallow /

peval 1.4b

Rule Path
Disallow /

pgp key agent

Rule Path
Disallow /

phantom

Rule Path
Disallow /

php/3.0.16

Rule Path
Disallow /

php/4.0.4pl1

Rule Path
Disallow /

php/4.0.6

Rule Path
Disallow /

php/4.1.1

Rule Path
Disallow /

php/4.1.2

Rule Path
Disallow /

phpdig

Rule Path
Disallow /

picosearch/1.0

Rule Path
Disallow /

piltdownman

Rule Path
Disallow /

pimptrain.com's robot

Rule Path
Disallow /

pingalink monitoring services 1.0

Rule Path
Disallow /

pingalink monitoring services 1.0 (http://www.pingalink.com)

Rule Path
Disallow /

pioneer

Rule Path
Disallow /

pizilla++ ver 2.45

Rule Path
Disallow /

pjspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)

Rule Path
Disallow /

plantynet_webrobot_v1.9 dhkang@plantynet.com

Rule Path
Disallow /

plumtreewebaccessor

Rule Path
Disallow /

pompos/1.0 pompos@iliad.fr

Rule Path
Disallow /

poppi

Rule Path
Disallow /

popular iconoclast

Rule Path
Disallow /

portal juice spider

Rule Path
Disallow /

portalb spider

Rule Path
Disallow /

processor/0.0alpha libwww-perl/0.20

Rule Path
Disallow /

progressive download

Rule Path
Disallow /

progressive download http check

Rule Path
Disallow /

proxy server

Rule Path
Disallow /

psbot

Rule Path
Disallow /

psbot/0.1 (+http://www.picsearch.com/bot.html)

Rule Path
Disallow /

psurf15a 11

Rule Path
Disallow /

psurf15a 51

Rule Path
Disallow /

psurf15a va

Rule Path
Disallow /

puresight

Rule Path
Disallow /

python-urllib/2.0a1

Rule Path
Disallow /

quarterdecks webcompass search tool

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

rainbot1.1

Rule Path
Disallow /

rambot xtreme x.x

Rule Path
Disallow /

raven search

Rule Path
Disallow /

rbse spider

Rule Path
Disallow /

rcn.com user agent ?

Rule Path
Disallow /

realdownload/4.0.0.4x

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

resume robot

Rule Path
Disallow /

rma/1.0 (compatible; realmedia)

Rule Path
Disallow /

road runner: the imagescape robot

Rule Path
Disallow /

roadhouse crawling system

Rule Path
Disallow /

robbie the robot

Rule Path
Disallow /

robocrawl spider

Rule Path
Disallow /

robofox

Rule Path
Disallow /

robot francoroute

Rule Path
Disallow /

robot@supersnooper.com

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

robozilla/1.0

Rule Path
Disallow /

rotondo/3.1 libwww/5.3.1

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

rpt-httpclient/0.3-x

Rule Path
Disallow /

rrc (crawler_admin@bigfoot.com)

Rule Path
Disallow /

rrc crawler_admin@bigfoot.com

Rule Path
Disallow /

rsurf15a 41

Rule Path
Disallow /

rsurf15a 51

Rule Path
Disallow /

rsurf15a 81

Rule Path
Disallow /

rules

Rule Path
Disallow /

safetynet robot

Rule Path
Disallow /

sandbot

Rule Path
Disallow /

science traveller international 1x/1.0

Rule Path
Disallow /

scooter

Rule Path
Disallow /

scooter/1.0 scooter@pa.dec.com

Rule Path
Disallow /

scooter/1.1 (custom)

Rule Path
Disallow /

scooter/2.0 g.r.a.b. v1.1.0

Rule Path
Disallow /

scooter/2.0 g.r.a.b. x2.0

Rule Path
Disallow /

scooter_bh0-3.0.3

Rule Path
Disallow /

scooter_trk3-3.0.3

Rule Path
Disallow /

scooter2_mercator_x-x.0

Rule Path
Disallow /

scooter-3.0.eu

Rule Path
Disallow /

scooter-3.0.fs

Rule Path
Disallow /

scooter-3.0.hd

Rule Path
Disallow /

scooter-3.0.vns

Rule Path
Disallow /

scooter-3.0qi

Rule Path
Disallow /

scooter-3.2

Rule Path
Disallow /

scooter-3.2.dil

Rule Path
Disallow /

scooter-3.2.ex

Rule Path
Disallow /

scooter-3.2.jt

Rule Path
Disallow /

scooter-3.2.niv

Rule Path
Disallow /

scooter-3.2.snippet

Rule Path
Disallow /

scooter-3.3dev

Rule Path
Disallow /

scooter-ars-1.1-ih

Rule Path
Disallow /

scooter-venus-3.0.vns

Rule Path
Disallow /

scooter-w3.1.2

Rule Path
Disallow /

scooter-w3-1.0

Rule Path
Disallow /

scoutabout

Rule Path
Disallow /

search.aus-au.com

Rule Path
Disallow /

search.ch v1.4

Rule Path
Disallow /

search.ch v1.4.2 (spiderman@search.ch; http://www.search.ch)

Rule Path
Disallow /

search+

Rule Path
Disallow /

searchexpress spider0.99

Rule Path
Disallow /

searchprocess

Rule Path
Disallow /

searchtone2.0 - ideare

Rule Path
Disallow /

secure-netz.de link checking (in conjunction w. lwp::simple/5.50)

Rule Path
Disallow /

senrigan

Rule Path
Disallow /

sg-scout

Rule Path
Disallow /

shagseeker

Rule Path
Disallow /

shai'hulud

Rule Path
Disallow /

sift

Rule Path
Disallow /

simmany robot ver1.0

Rule Path
Disallow /

site searcher

Rule Path
Disallow /

site valet

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitecheck.internetseer.com (for more info see: http://sitecheck.internetseer.

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitetech-rover

Rule Path
Disallow /

skymob.com

Rule Path
Disallow /

slarp/0.1

Rule Path
Disallow /

slcrawler

Rule Path
Disallow /

sleek

Rule Path
Disallow /

slider_search_v1-de

Rule Path
Disallow /

slowbot

Rule Path
Disallow /

slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slurp/2.0-kiteweekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slysearch (slysearch@slysearch.com)

Rule Path
Disallow /

smart spider

Rule Path
Disallow /

smartdownload/1.2.67 (win32; jan 12 1999)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; feb 1 2000)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; jun 19 2001)

Rule Path
Disallow /

snooper

Rule Path
Disallow /

solbot

Rule Path
Disallow /

some download agent, in conjunction w. "dlman"

Rule Path
Disallow /

some download agent, in conjunction w. "wildsoft surfer"

Rule Path
Disallow /

some download manager

Rule Path
Disallow /

some download manager (spoofed user-agent)

Rule Path
Disallow /

some download manager spoofing as ms frontpage 3.x

Rule Path
Disallow /

some download manager spoofing netscape 4.01

Rule Path
Disallow /

some private robot (wanadoo.fr client)

Rule Path
Disallow /

spanner

Rule Path
Disallow /

speedfind rambot xtreme 8.1

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

spida/0.1

Rule Path
Disallow /

spider.batsch.com

Rule Path
Disallow /

spider.yellopet.com - www.yellopet.com

Rule Path
Disallow /

spider/maxbot.com admin@maxbot.com

Rule Path
Disallow /

spider_monkey

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

spiderline crawler

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

spiderview(tm)

Rule Path
Disallow /

spinne/2.0 med

Rule Path
Disallow /

spry wizard robot

Rule Path
Disallow /

spyder3.microsys.com

Rule Path
Disallow /

sq webscanner

Rule Path
Disallow /

sqworm/2.9.81-beta (beta_release; 20011102-760; i686-pc-linux-gnu)

Rule Path
Disallow /

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux-gnu)

Rule Path
Disallow /

sqworm/2.9.89-beta (beta_release; 20020130-839; i686-pc-linux-gnu)

Rule Path
Disallow /

ssurf15a 11

Rule Path
Disallow /

stamina/1.4

Rule Path
Disallow /

star downloader

Rule Path
Disallow /

statfetcher/1.0

Rule Path
Disallow /

suke

Rule Path
Disallow /

summycrawler

Rule Path
Disallow /

suntek search engine

Rule Path
Disallow /

superbot/2.6 (win32)

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surferf3 1/0

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

sven

Rule Path
Disallow /

swbot/0.9c libwww/5.3.1

Rule Path
Disallow /

syncbot

Rule Path
Disallow /

syncit link validation

Rule Path
Disallow /

syncit/1.x

Rule Path
Disallow /

syncit/2.x

Rule Path
Disallow /

szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl

Rule Path
Disallow /

tach black widow

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

targetblaster.com/0.9k

Rule Path
Disallow /

tarspider

Rule Path
Disallow /

tcl w3 robot

Rule Path
Disallow /

te

Rule Path
Disallow /

teachersoft/1.0 libwww/2.17

Rule Path
Disallow /

teamsoft wininet component

Rule Path
Disallow /

techbot

Rule Path
Disallow /

telefonica.es user robot

Rule Path
Disallow /

teleport pro/1.29.1xxx

Rule Path
Disallow /

templeton

Rule Path
Disallow /

teomaagent crawler-admin@teoma.com

Rule Path
Disallow /

teomaagent1 [crawler-admin@teoma.com]

Rule Path
Disallow /

teomatechnologies

Rule Path
Disallow /

the expert html source viewer (http://www.expert-html.com)

Rule Path
Disallow /

the jubii indexing robot

Rule Path
Disallow /

the northstar robot

Rule Path
Disallow /

the nwi robot

Rule Path
Disallow /

the peregrinator

Rule Path
Disallow /

the python robot

Rule Path
Disallow /

the tkwww robot

Rule Path
Disallow /

the web moose

Rule Path
Disallow /

the web wombat

Rule Path
Disallow /

the webfoot robot

Rule Path
Disallow /

the world wide web wanderer

Rule Path
Disallow /

the world wide web worm

Rule Path
Disallow /

time warner telecom user robot ?

Rule Path
Disallow /

titan

Rule Path
Disallow /

titin

Rule Path
Disallow /

tivraspider/1.0 (crawler@tivra.com)

Rule Path
Disallow /

tjg/spider

Rule Path
Disallow /

tlspider

Rule Path
Disallow /

toutatis 2.5-2

Rule Path
Disallow /

trampelpfad-spider

Rule Path
Disallow /

trampelpfad-spider-v0.1

Rule Path
Disallow /

tricosmetacheck 1.2216-08-1999 (http://www.tricos.com/metacheck)

Rule Path
Disallow /

tsurf15a 11

Rule Path
Disallow /

twotrees reactive filter v2.0

Rule Path
Disallow /

ucmore

Rule Path
Disallow /

ucmore crawler app

Rule Path
Disallow /

ucsd crawl

Rule Path
Disallow /

udm

Rule Path
Disallow /

udmsearch

Rule Path
Disallow /

udmsearch/3.1.x

Rule Path
Disallow /

uiowacrawler/1.0

Rule Path
Disallow /

ultimate search / smartdesk (no website) robot

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

unet.ca

Rule Path
Disallow /

unknown link-checking

Rule Path
Disallow /

unknown robots from 64.5.216.xx, 66.98.36.xx, 148.74.255.xx

Rule Path
Disallow /

unknown user robot (24.124.34.42)

Rule Path
Disallow /

unknown.level3.net robot

Rule Path
Disallow /

unnamed

Rule Path
Disallow /

up.browser/3.01-ig01 up.link/3.2.3.4

Rule Path
Disallow /

url check

Rule Path
Disallow /

url spider pro

Rule Path
Disallow /

urlgetfile

Rule Path
Disallow /

user agent

Rule Path
Disallow /

user agent (http header f. transferencoding)

Rule Path
Disallow /

user agent, looks for form-mail components (spam-bot)

Rule Path
Disallow /

user faked referer

Rule Path
Disallow /

utilmind httpget

Rule Path
Disallow /

utopia webwasher 3.0

Rule Path
Disallow /

uu.net user robot/agent

Rule Path
Disallow /

vagabondo/1.x mt (webagent@wise-guys.nl)

Rule Path
Disallow /

valkyrie

Rule Path
Disallow /

verticrawl

Rule Path
Disallow /

victoria

Rule Path
Disallow /

vision-search

Rule Path
Disallow /

voyager

Rule Path
Disallow /

vspider/3.x

Rule Path
Disallow /

vwbot

Rule Path
Disallow /

w@pspider by wap4.com

Rule Path
Disallow /

w3c_validator/1.122 libwww-perl/5.50

Rule Path
Disallow /

w3index

Rule Path
Disallow /

w3m/0.1.xx

Rule Path
Disallow /

w3m2

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

waldec.com

Rule Path
Disallow /

walhello appie

Rule Path
Disallow /

wallpaper

Rule Path
Disallow /

wanadoo internet provider

Rule Path
Disallow /

wanadoo internet services

Rule Path
Disallow /

wdg_validator/1.1

Rule Path
Disallow /

web core / roots

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

web link validator 1.5

Rule Path
Disallow /

webbandit web spider

Rule Path
Disallow /

webbandit/4.45.0

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webcollage/1.7x

Rule Path
Disallow /

webcollage/1.8x

Rule Path
Disallow /

webcollector

Rule Path
Disallow /

webcompass 2.0

Rule Path
Disallow /

webcopier v2.5

Rule Path
Disallow /

webcopier v2.7a

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

webfindbot(http://www.web-find.com)

Rule Path
Disallow /

webhack

Rule Path
Disallow /

weblayers

Rule Path
Disallow /

weblinker

Rule Path
Disallow /

weblink's checker/

Rule Path
Disallow /

weblog monitor

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webquest

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webreaper vx.x - www.webreaper.net

Rule Path
Disallow /

webs

Rule Path
Disallow /

websnarf

Rule Path
Disallow /

webspider

Rule Path
Disallow /

webspinne/1.0 webmaster@webspinne.de

Rule Path
Disallow /

webstolperer

Rule Path
Disallow /

webstripper/2.0x

Rule Path
Disallow /

webtrends/3.0 (winnt)

Rule Path
Disallow /

webtv

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webwalker

Rule Path
Disallow /

webwatch

Rule Path
Disallow /

webzinger

Rule Path
Disallow /

webzip/4.1 (http://www.spidersoft.com)

Rule Path
Disallow /

wfarc

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget/1.x

Rule Path
Disallow /

whatuseek winona

Rule Path
Disallow /

whatuseek_winona/3.0

Rule Path
Disallow /

whizbang! lab

Rule Path
Disallow /

whowhere robot

Rule Path
Disallow /

wild ferret web hopper

Product Comment
wild ferret web hopper 1, #2, #3
Rule Path
Disallow /

wildsoft surfer

Rule Path
Disallow /

willow internet crawler by twotrees v2.1

Rule Path
Disallow /

wind.it client user-agent ?

Rule Path
Disallow /

windows me, btopenworld internet services

Rule Path
Disallow /

windows me, internet explorer url check

Rule Path
Disallow /

winme, opera 5.x

Rule Path
Disallow /

winxp

Rule Path
Disallow /

wired digital

Rule Path
Disallow /

wisebot/1.0 (wisebot@koreawisenut.com; http://wisebot.koreawisenut.com)

Rule Path
Disallow /

wisewire-spider2

Rule Path
Disallow /

wobot/1.0 from 206.214.202.45

Rule Path
Disallow /

www collector

Rule Path
Disallow /

www.arianna.it

Rule Path
Disallow /

www.freeloader.com.

Rule Path
Disallow /

www2000.ogsm.vanderbilt.edu

Rule Path
Disallow /

wwwc ver 0.2.5

Rule Path
Disallow /

wwwoffle/2.x

Rule Path
Disallow /

xenu link sleuth 1.xx

Rule Path
Disallow /

xenu's link sleuth 1.1a

Rule Path
Disallow /

xget

Rule Path
Disallow /

xpc-mii.net http server message

Rule Path
Disallow /

xyleme robot

Rule Path
Disallow /

xyro_(xcrawler@cosmos.inria.fr)

Rule Path
Disallow /

zbot/1.00 (icaulfield@zeus.com)

Rule Path
Disallow /

zeus 1680 webster pro v2.9 win32

Rule Path
Disallow /

zeus 37389 webster pro v2.9 win32

Rule Path
Disallow /

zeus 86122 webster pro v2.9 win32

Rule Path
Disallow /

zeus themesite viewer webster pro v2.9 win32

Rule Path
Disallow /

zippy v2.0 - zippyfinder.com

Rule Path
Disallow /

zyborg/1.0 (zyborg@wisenut.com; http://www.wisenut.com)

Rule Path
Disallow /
Back to top

Warnings

  • 14 invalid lines.
Back to top