Log In Sign Up

techspotoronto.ca
robots.txt

Robots Exclusion Standard data for techspotoronto.ca

Resource Scan

Scan Details

Site Domain techspotoronto.ca
Base Domain techspotoronto.ca
Scan Status Ok
Last Scan4/8/2025, 4:41:20 AM
Next Scan 5/8/2025, 4:41:20 AM

Last Scan

Scanned4/8/2025, 4:41:20 AM
URL https://techspotoronto.ca/robots.txt
Domain IPs 104.21.68.72, 172.67.191.170, 2606:4700:3030::ac43:bfaa, 2606:4700:3031::6815:4448
Response IP 172.67.191.170
Found Yes
Hash 18403017c256f6d28c76ec455265dcf7f9fc70c9332857c26b3e97865a498591
SimHash 63bae7f50f07

Groups

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/

Other Records

Field Value
crawl-delay 500

googlebot

Rule Path
Disallow /wp-admin/
Allow /wp-includes/
Allow /wp-content/plugins/
Allow /wp-content/themes/
Allow /.js%24*
Allow /.inc%24*
Allow /.css%24*
Allow /.php%24*

Other Records

Field Value
crawl-delay 500

googlebot-mobile

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/

Other Records

Field Value
crawl-delay 500

googlebot-image

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/

Other Records

Field Value
crawl-delay 500

slurp

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/

Other Records

Field Value
crawl-delay 500

bingbot

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/

Other Records

Field Value
crawl-delay 300

mediapartners-google

Rule Path
Allow /

abac

Rule Path
Disallow /

abont

Rule Path
Disallow /

abot

Rule Path
Disallow /

accept

Rule Path
Disallow /

access

Rule Path
Disallow /

accoo

Rule Path
Disallow /

aceftp

Rule Path
Disallow /

acme

Rule Path
Disallow /

activetouristbot

Rule Path
Disallow /

address

Rule Path
Disallow /

adopt

Rule Path
Disallow /

adress

Rule Path
Disallow /

adressendeutschland

Rule Path
Disallow /

adsarobot

Rule Path
Disallow /

aesop\_com\_spiderman

Rule Path
Disallow /

agent

Rule Path
Disallow /

ah\-ha

Rule Path
Disallow /

ahead

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot/1.0

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

akregat

Rule Path
Disallow /

aktuelles

Rule Path
Disallow /

alarm

Rule Path
Disallow /

albert

Rule Path
Disallow /

alek

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alligator

Rule Path
Disallow /

allsubmitter

Rule Path
Disallow /

alma

Rule Path
Disallow /

almaden

Rule Path
Disallow /

alot

Rule Path
Disallow /

alpha

Rule Path
Disallow /

amfi

Rule Path
Disallow /

amzn\_assoc

Rule Path
Disallow /

anal

Rule Path
Disallow /

anarchie

Rule Path
Disallow /

andit

Rule Path
Disallow /

anon

Rule Path
Disallow /

anotherbot

Rule Path
Disallow /

ansearch

Rule Path
Disallow /

answerbus

Rule Path
Disallow /

antivirx

Rule Path
Disallow /

apexoo

Rule Path
Disallow /

appie

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

archive

Rule Path
Disallow /

arian

Rule Path
Disallow /

aspse

Rule Path
Disallow /

assort

Rule Path
Disallow /

aster

Rule Path
Disallow /

asterias

Rule Path
Disallow /

atari

Rule Path
Disallow /

athens

Rule Path
Disallow /

athome

Rule Path
Disallow /

atlocal

Rule Path
Disallow /

atomic_email_hunter

Rule Path
Disallow /

atomz

Rule Path
Disallow /

atrop

Rule Path
Disallow /

attrib

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

autohttp

Rule Path
Disallow /

axod

Rule Path
Disallow /

b2w

Rule Path
Disallow /

back

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backstreet

Rule Path
Disallow /

backweb

Rule Path
Disallow /

badass

Rule Path
Disallow /

baid

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

bali

Rule Path
Disallow /

bandit

Rule Path
Disallow /

barry

Rule Path
Disallow /

basichttp

Rule Path
Disallow /

batch

Rule Path
Disallow /

batchftp

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

beat

Rule Path
Disallow /

become

Rule Path
Disallow /

beij

Rule Path
Disallow /

benchmark

Rule Path
Disallow /

berts

Rule Path
Disallow /

bew

Rule Path
Disallow /

big.brother

Rule Path
Disallow /

bigfoot

Rule Path
Disallow /

bilgi

Rule Path
Disallow /

bison

Rule Path
Disallow /

bitacle

Rule Path
Disallow /

biz360

Rule Path
Disallow /

black

Rule Path
Disallow /

black hole

Rule Path
Disallow /

black.hole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bladder.fusion

Rule Path
Disallow /

blaiz

Rule Path
Disallow /

blog.checker

Rule Path
Disallow /

blogl

Rule Path
Disallow /

blogpeople

Rule Path
Disallow /

blogshares.spiders

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

blow

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

bmclient

Rule Path
Disallow /

board

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

boi

Rule Path
Disallow /

boitho

Rule Path
Disallow /

bolt\ 0

Rule Path
Disallow /

bond

Rule Path
Disallow /

bookmark.search.tool

Rule Path
Disallow /

boris

Rule Path
Disallow /

bost

Rule Path
Disallow /

boston.project

Rule Path
Disallow /

bot.mailto:craftbot@yahoo.com

Rule Path
Disallow /

botalot

Rule Path
Disallow /

botpaidtoclick

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

botw

Rule Path
Disallow /

brandwatch

Rule Path
Disallow /

bravobrian

Rule Path
Disallow /

brok

Rule Path
Disallow /

bropwers

Rule Path
Disallow /

broth

Rule Path
Disallow /

browseabit

Rule Path
Disallow /

browsex

Rule Path
Disallow /

browsezilla

Rule Path
Disallow /

bruin

Rule Path
Disallow /

bsalsa

Rule Path
Disallow /

buddy

Rule Path
Disallow /

build

Rule Path
Disallow /

built

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bulls

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

bumblebee

Rule Path
Disallow /

bunny

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

busca

Rule Path
Disallow /

busi

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

buy

Rule Path
Disallow /

bwh3

Rule Path
Disallow /

c\-spider

Rule Path
Disallow /

cafek

Rule Path
Disallow /

cafi

Rule Path
Disallow /

camel

Rule Path
Disallow /

cand

Rule Path
Disallow /

captu

Rule Path
Disallow /

catch

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cc-rget/5.818

Rule Path
Disallow /

cd34

Rule Path
Disallow /

ceg

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

cgichk

Rule Path
Disallow /

cha0s

Rule Path
Disallow /

chang

Rule Path
Disallow /

chaos

Rule Path
Disallow /

char

Rule Path
Disallow /

char\(32\,35\)

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

chek

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

chill

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

cicc

Rule Path
Disallow /

cisco

Rule Path
Disallow /

cita

Rule Path
Disallow /

clam

Rule Path
Disallow /

claw

Rule Path
Disallow /

click.bot

Rule Path
Disallow /

clipping

Rule Path
Disallow /

clshttp

Rule Path
Disallow /

clush

Rule Path
Disallow /

coast

Rule Path
Disallow /

coldfusion

Rule Path
Disallow /

coll

Rule Path
Disallow /

comb

Rule Path
Disallow /

commentreader

Rule Path
Disallow /

compan

Rule Path
Disallow /

conc

Rule Path
Disallow /

contact

Rule Path
Disallow /

control

Rule Path
Disallow /

contype

Rule Path
Disallow /

conv

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copi

Rule Path
Disallow /

copy

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

coral

Rule Path
Disallow /

core-project

Rule Path
Disallow /

corn

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

costa

Rule Path
Disallow /

cr4nk

Rule Path
Disallow /

craft

Rule Path
Disallow /

crank

Rule Path
Disallow /

crap

Rule Path
Disallow /

crawler0

Rule Path
Disallow /

crazy

Rule Path
Disallow /

cres

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

cs\-cz

Rule Path
Disallow /

cshttp

Rule Path
Disallow /

cuill

Rule Path
Disallow /

curl

Rule Path
Disallow /

custo

Rule Path
Disallow /

cute

Rule Path
Disallow /

cyber

Rule Path
Disallow /

cyberalert

Rule Path
Disallow /

daobot

Rule Path
Disallow /

dark

Rule Path
Disallow /

data

Rule Path
Disallow /

daten

Rule Path
Disallow /

daum

Rule Path
Disallow /

dcbot

Rule Path
Disallow /

dcs

Rule Path
Disallow /

deep

Rule Path
Disallow /

default\ browser\ 0

Rule Path
Disallow /

deps

Rule Path
Disallow /

detect

Rule Path
Disallow /

deweb

Rule Path
Disallow /

diam

Rule Path
Disallow /

digger

Rule Path
Disallow /

digimarc

Rule Path
Disallow /

digout4uagent

Rule Path
Disallow /

diibot

Rule Path
Disallow /

dillo

Rule Path
Disallow /

ding

Rule Path
Disallow /

disc

Rule Path
Disallow /

disco

Rule Path
Disallow /

discobot

Rule Path
Disallow /

discobot/1.0

Rule Path
Disallow /

disp

Rule Path
Disallow /

ditto

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dlc

Rule Path
Disallow /

dnloadmage

Rule Path
Disallow /

dns-digger/1.0

Rule Path
Disallow /

dns-digger-explorer/1.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

doubanbot

Rule Path
Disallow /

download

Rule Path
Disallow /

download.demon

Rule Path
Disallow /

download.devil

Rule Path
Disallow /

download.wonder

Rule Path
Disallow /

download\ demon

Rule Path
Disallow /

downloader

Rule Path
Disallow /

drag

Rule Path
Disallow /

dreampassport

Rule Path
Disallow /

drec

Rule Path
Disallow /

drip

Rule Path
Disallow /

dsdl

Rule Path
Disallow /

dsok

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

dtaagent

Rule Path
Disallow /

dts

Rule Path
Disallow /

dual

Rule Path
Disallow /

dumb

Rule Path
Disallow /

dynaweb

Rule Path
Disallow /

e\-collector

Rule Path
Disallow /

eag

Rule Path
Disallow /

earn

Rule Path
Disallow /

earthcom

Rule Path
Disallow /

easydl

Rule Path
Disallow /

ebin

Rule Path
Disallow /

ebm-apple

Rule Path
Disallow /

ebrowse

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

echo

Rule Path
Disallow /

ecollector

Rule Path
Disallow /

ecxi

Rule Path
Disallow /

edco

Rule Path
Disallow /

edgeio

Rule Path
Disallow /

efp\@gmx\.net

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

email

Rule Path
Disallow /

email.extractor

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsearch

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

emer

Rule Path
Disallow /

empas

Rule Path
Disallow /

enfi

Rule Path
Disallow /

enhan

Rule Path
Disallow /

enterprise\_search

Rule Path
Disallow /

envolk

Rule Path
Disallow /

erck

Rule Path
Disallow /

erocr

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

esurf

Rule Path
Disallow /

eval

Rule Path
Disallow /

evere

Rule Path
Disallow /

evil

Rule Path
Disallow /

ewh

Rule Path
Disallow /

exabot

Rule Path
Disallow /

exabot/2.0

Rule Path
Disallow /

exact

Rule Path
Disallow /

exploiter

Rule Path
Disallow /

expre

Rule Path
Disallow /

express\ webpictures

Rule Path
Disallow /

extra

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyen

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fairad

Rule Path
Disallow /

fake

Rule Path
Disallow /

fang

Rule Path
Disallow /

fast

Rule Path
Disallow /

fastlwspider

Rule Path
Disallow /

favorg

Rule Path
Disallow /

favorites.sweeper

Rule Path
Disallow /

faxo

Rule Path
Disallow /

fdm\_1

Rule Path
Disallow /

fdse

Rule Path
Disallow /

fetch

Rule Path
Disallow /

fezhead

Rule Path
Disallow /

filan

Rule Path
Disallow /

filehound

Rule Path
Disallow /

find

Rule Path
Disallow /

firebat

Rule Path
Disallow /

firefox.2\.0

Rule Path
Disallow /

firs

Rule Path
Disallow /

flam

Rule Path
Disallow /

flash

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flickbot

Rule Path
Disallow /

flip

Rule Path
Disallow /

fluffy

Rule Path
Disallow /

flunky

Rule Path
Disallow /

focus

Rule Path
Disallow /

foob

Rule Path
Disallow /

foobot

Rule Path
Disallow /

fooky

Rule Path
Disallow /

forex

Rule Path
Disallow /

forum

Rule Path
Disallow /

forv

Rule Path
Disallow /

fost

Rule Path
Disallow /

foto

Rule Path
Disallow /

foun

Rule Path
Disallow /

franklin.locator

Rule Path
Disallow /

freefind

Rule Path
Disallow /

freshdownload

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

fsurf

Rule Path
Disallow /

fuck

Rule Path
Disallow /

fuer

Rule Path
Disallow /

futile

Rule Path
Disallow /

fyber

Rule Path
Disallow /

gais

Rule Path
Disallow /

galaxybot

Rule Path
Disallow /

galbot

Rule Path
Disallow /

gamespy\_arcade

Rule Path
Disallow /

gbpl

Rule Path
Disallow /

gener

Rule Path
Disallow /

geni

Rule Path
Disallow /

geona

Rule Path
Disallow /

get

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

gigabaz

Rule Path
Disallow /

ginxbot

Rule Path
Disallow /

gira

Rule Path
Disallow /

gluc

Rule Path
Disallow /

glx.?v

Rule Path
Disallow /

gnome

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go.zilla

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

goforit

Rule Path
Disallow /

goldfire

Rule Path
Disallow /

gonzo

Rule Path
Disallow /

gornker

Rule Path
Disallow /

gosearch

Rule Path
Disallow /

got\-it

Rule Path
Disallow /

gozilla

Rule Path
Disallow /

grab

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

graf

Rule Path
Disallow /

grafula

Rule Path
Disallow /

green.research

Rule Path
Disallow /

grub

Rule Path
Disallow /

grub\-client

Rule Path
Disallow /

grup

Rule Path
Disallow /

gsa\-cra

Rule Path
Disallow /

gsearch

Rule Path
Disallow /

gt::www

Rule Path
Disallow /

gt\:\:www

Rule Path
Disallow /

guidebot

Rule Path
Disallow /

guruji

Rule Path
Disallow /

gvfs

Rule Path
Disallow /

gyps

Rule Path
Disallow /

hack

Rule Path
Disallow /

haha

Rule Path
Disallow /

hailo

Rule Path
Disallow /

harv

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

hatena

Rule Path
Disallow /

hax

Rule Path
Disallow /

head

Rule Path
Disallow /

helm

Rule Path
Disallow /

herit

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hgre

Rule Path
Disallow /

hhjhj\@yahoo

Rule Path
Disallow /

hippo

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hmse

Rule Path
Disallow /

hmview

Rule Path
Disallow /

holm

Rule Path
Disallow /

holy

Rule Path
Disallow /

homepagesearch

Rule Path
Disallow /

hoowwwer

Rule Path
Disallow /

houxoucrawler

Rule Path
Disallow /

hpprint

Rule Path
Disallow /

htdig

Rule Path
Disallow /

http.generic

Rule Path
Disallow /

http::lite

Rule Path
Disallow /

httpconnect

Rule Path
Disallow /

httpdown

Rule Path
Disallow /

httpget

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httpretriever

Rule Path
Disallow /

httrack

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

human

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

huron

Rule Path
Disallow /

hverify

Rule Path
Disallow /

hybrid

Rule Path
Disallow /

hyper

Rule Path
Disallow /

ia\_archiver

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

iaskspi

Rule Path
Disallow /

ibm\_planetwide

Rule Path
Disallow /

iccra

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

id\-search

Rule Path
Disallow /

ida

Rule Path
Disallow /

idbot

Rule Path
Disallow /

id-search

Rule Path
Disallow /

id-search\.org

Rule Path
Disallow /

ieauto

Rule Path
Disallow /

iempt

Rule Path
Disallow /

iexplore\.exe

Rule Path
Disallow /

igetter

Rule Path
Disallow /

iimds\_monitor

Rule Path
Disallow /

ilse

Rule Path
Disallow /

iltrov

Rule Path
Disallow /

image

Rule Path
Disallow /

image.stripper

Rule Path
Disallow /

image.sucker

Rule Path
Disallow /

image\ stripper

Rule Path
Disallow /

image\ sucker

Rule Path
Disallow /

imagefetch

Rule Path
Disallow /

incutio

Rule Path
Disallow /

incywincy

Rule Path
Disallow /

indexer

Rule Path
Disallow /

industry.program

Rule Path
Disallow /

indy

Rule Path
Disallow /

indy\ library

Rule Path
Disallow /

ineturl

Rule Path
Disallow /

infonav

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

informant

Rule Path
Disallow /

infotekies

Rule Path
Disallow /

ingelin

Rule Path
Disallow /

innerpr

Rule Path
Disallow /

inspect

Rule Path
Disallow /

installshield.digitalwizard

Rule Path
Disallow /

insuran\.

Rule Path
Disallow /

intellig

Rule Path
Disallow /

intelliseek

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet.ninja

Rule Path
Disallow /

internet.x

Rule Path
Disallow /

internet\ ninja

Rule Path
Disallow /

internet\_explorer

Rule Path
Disallow /

internetlinkagent

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

internetseer\.com

Rule Path
Disallow /

intraf

Rule Path
Disallow /

ip2

Rule Path
Disallow /

ipsel

Rule Path
Disallow /

iria

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

iron33

Rule Path
Disallow /

irvine

Rule Path
Disallow /

isc\ systems\ irc\ search\ 2\.1

Rule Path
Disallow /

isc\_sys

Rule Path
Disallow /

isilo

Rule Path
Disallow /

isrccrawler

Rule Path
Disallow /

isspi

Rule Path
Disallow /

iupui.research.bot

Rule Path
Disallow /

jady

Rule Path
Disallow /

jaka

Rule Path
Disallow /

jakarta

Rule Path
Disallow /

jakarta commons-httpclient/3.0

Rule Path
Disallow /

jam

Rule Path
Disallow /

java

Rule Path
Disallow /

java\(tm\)

Rule Path
Disallow /

java\/

Rule Path
Disallow /

jbh.agent

Rule Path
Disallow /

jenny

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetb

Rule Path
Disallow /

jetc

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jeteye

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

jiro

Rule Path
Disallow /

jobo

Rule Path
Disallow /

joc

Rule Path
Disallow /

joc\ web\ spider

Rule Path
Disallow /

jupit

Rule Path
Disallow /

just

Rule Path
Disallow /

jyx

Rule Path
Disallow /

kapere

Rule Path
Disallow /

kash

Rule Path
Disallow /

kazo

Rule Path
Disallow /

kbee

Rule Path
Disallow /

kenjin

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

kernel

Rule Path
Disallow /

keywo

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

kfsw

Rule Path
Disallow /

kkma

Rule Path
Disallow /

know

Rule Path
Disallow /

kosmix

Rule Path
Disallow /

krae

Rule Path
Disallow /

kretrieve

Rule Path
Disallow /

krug

Rule Path
Disallow /

ksibot

Rule Path
Disallow /

ksoap

Rule Path
Disallow /

kum

Rule Path
Disallow /

kwebget

Rule Path
Disallow /

lachesis

Rule Path
Disallow /

lanshan

Rule Path
Disallow /

lapo

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leacher

Rule Path
Disallow /

leech

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

leechget

Rule Path
Disallow /

leipzig\.de

Rule Path
Disallow /

lets

Rule Path
Disallow /

lexi

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lftp

Rule Path
Disallow /

libby

Rule Path
Disallow /

libcrawl

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

libfetch

Rule Path
Disallow /

libghttp

Rule Path
Disallow /

libweb

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

libwhisker

Rule Path
Disallow /

libwww

Rule Path
Disallow /

libwww\-fm

Rule Path
Disallow /

libwww\-perl

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

libwww-perl/5.813

Rule Path
Disallow /

lightningdownload

Rule Path
Disallow /

likse

Rule Path
Disallow /

linc

Rule Path
Disallow /

link

Rule Path
Disallow /

link.sleuth

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkie

Rule Path
Disallow /

links.aromatized

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkscan/8.1a unix

Rule Path
Disallow /

linksmanager.com_bot

Rule Path
Disallow /

linktiger

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lint

Rule Path
Disallow /

list

Rule Path
Disallow /

lmcrawler

Rule Path
Disallow /

lmq

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

loader

Rule Path
Disallow /

localcombot

Rule Path
Disallow /

locu

Rule Path
Disallow /

london

Rule Path
Disallow /

lone

Rule Path
Disallow /

looksmart

Rule Path
Disallow /

loop

Rule Path
Disallow /

lork

Rule Path
Disallow /

lth\_

Rule Path
Disallow /

lwp

Rule Path
Disallow /

lwp\-request

Rule Path
Disallow /

lwp-request

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

lwp-trivial/1.34

Rule Path
Disallow /

mac.finder

Rule Path
Disallow /

mac\_f

Rule Path
Disallow /

macintosh\;.i\;.ppc

Rule Path
Disallow /

mag\-net

Rule Path
Disallow /

magi

Rule Path
Disallow /

magnet

Rule Path
Disallow /

magp

Rule Path
Disallow /

magpie-crawler 1.1

Rule Path
Disallow /

mail.sweeper

Rule Path
Disallow /

main

Rule Path
Disallow /

majest

Rule Path
Disallow /

mam

Rule Path
Disallow /

mana

Rule Path
Disallow /

marcopolo

Rule Path
Disallow /

mark.blonin

Rule Path
Disallow /

markwatch

Rule Path
Disallow /

masagool

Rule Path
Disallow /

mass

Rule Path
Disallow /

mass.downloader

Rule Path
Disallow /

mass\ downloader

Rule Path
Disallow /

mata

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mavi

Rule Path
Disallow /

maxthon$

Rule Path
Disallow /

mcbot

Rule Path
Disallow /

mcspider

Rule Path
Disallow /

mecha

Rule Path
Disallow /

mediapartners

Rule Path
Disallow /

megaupload

Rule Path
Disallow /

metaproducts.download.express

Rule Path
Disallow /

metaspin

Rule Path
Disallow /

mete

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

microsoft url control - 5.01.4511

Rule Path
Disallow /

microsoft url control - 6.00.8169

Rule Path
Disallow /

microsoft.data.access

Rule Path
Disallow /

microsoft.url

Rule Path
Disallow /

microsoft\ url\ control

Rule Path
Disallow /

microsoft\.url

Rule Path
Disallow /

microsoft\_internet\_explorer

Rule Path
Disallow /

mido

Rule Path
Disallow /

midown\ tool

Rule Path
Disallow /

miix

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

miixpc/4.2

Rule Path
Disallow /

miner

Rule Path
Disallow /

mira

Rule Path
Disallow /

mire

Rule Path
Disallow /

mirror

Rule Path
Disallow /

miss

Rule Path
Disallow /

missauga

Rule Path
Disallow /

missigua.locator

Rule Path
Disallow /

missigua\ locator

Rule Path
Disallow /

missouri.college.browse

Rule Path
Disallow /

mist

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mister\ pix

Rule Path
Disallow /

mizz

Rule Path
Disallow /

mj12

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.2.0

Rule Path
Disallow /

mj12bot/v1.2.5

Rule Path
Disallow /

mkdb

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

mlm

Rule Path
Disallow /

mmmocrawl

Rule Path
Disallow /

mnog

Rule Path
Disallow /

moge

Rule Path
Disallow /

moget

Rule Path
Disallow /

moget/2.1

Rule Path
Disallow /

moje

Rule Path
Disallow /

monster

Rule Path
Disallow /

monza.browser

Rule Path
Disallow /

mooz

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

moreoverbot/5.0

Rule Path
Disallow /

mot\-mpx220

Rule Path
Disallow /

mothra\/netscan

Rule Path
Disallow /

mouse

Rule Path
Disallow /

movabletype

Rule Path
Disallow /

mozdex

Rule Path
Disallow /

mozi\!

Rule Path
Disallow /

mozilla/4

Rule Path
Disallow /

mozilla/4.0 (compatible; bullseye; windows 95)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows 2000)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows 95)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows 98)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows me)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows nt)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 4.0; windows xp)

Rule Path
Disallow /

mozilla/5

Rule Path
Disallow /

mozilla\.*indy

Rule Path
Disallow /

mozilla\.*newt

Rule Path
Disallow /

mozilla\/1\.22

Rule Path
Disallow /

mozilla\/22

Rule Path
Disallow /

mozilla\/3\.mozilla\/2\.01

Rule Path
Disallow /

mozilla\/4\.0\(compatible

Rule Path
Disallow /

mozilla\/4\.08

Rule Path
Disallow /

mozilla\/4\.61.\(macintosh

Rule Path
Disallow /

mozilla\/5\.0

Rule Path
Disallow /

mozilla\/7\.0

Rule Path
Disallow /

mozilla\/8

Rule Path
Disallow /

mozilla\/9

Rule Path
Disallow /

mozilla\/firefox

Rule Path
Disallow /

mozilla\:

Rule Path
Disallow /

mp3bot

Rule Path
Disallow /

mpf

Rule Path
Disallow /

mra

Rule Path
Disallow /

ms.?search

Rule Path
Disallow /

ms.frontpage

Rule Path
Disallow /

msfrontpage

Rule Path
Disallow /

msie\_6\.0

Rule Path
Disallow /

msie6

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msnbot\-media

Rule Path
Disallow /

msnbot\-products

Rule Path
Disallow /

msnptc

Rule Path
Disallow /

msproxy

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

multithreaddb

Rule Path
Disallow /

musc

Rule Path
Disallow /

mvac

Rule Path
Disallow /

mwm

Rule Path
Disallow /

my\_age

Rule Path
Disallow /

myapp

Rule Path
Disallow /

mydog

Rule Path
Disallow /

myeng

Rule Path
Disallow /

myfamilybot

Rule Path
Disallow /

mygetright

Rule Path
Disallow /

myie2

Rule Path
Disallow /

mysearch

Rule Path
Disallow /

myurl

Rule Path
Disallow /

nag

Rule Path
Disallow /

nameprotect

Rule Path
Disallow /

nasa.search

Rule Path
Disallow /

nationaldirectory

Rule Path
Disallow /

naver

Rule Path
Disallow /

navr

Rule Path
Disallow /

navroad

Rule Path
Disallow /

near

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

net.vampire

Rule Path
Disallow /

net\ vampire

Rule Path
Disallow /

netants

Rule Path
Disallow /

netattache

Rule Path
Disallow /

netcach

Rule Path
Disallow /

netcarta

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netcrawl

Rule Path
Disallow /

netmech

Rule Path
Disallow /

netprospector

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

netseer crawler/2.0

Rule Path
Disallow /

netseer/0.1

Rule Path
Disallow /

netseer/nutch-0.9

Rule Path
Disallow /

netsp

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netx

Rule Path
Disallow /

netz

Rule Path
Disallow /

netzip

Rule Path
Disallow /

neut

Rule Path
Disallow /

newlisp

Rule Path
Disallow /

newsgatorinbox

Rule Path
Disallow /

newt

Rule Path
Disallow /

newt.activex

Rule Path
Disallow /

next

Rule Path
Disallow /

nice

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nikto

Rule Path
Disallow /

nimb

Rule Path
Disallow /

ninja

Rule Path
Disallow /

ninte

Rule Path
Disallow /

nipgcrawler

Rule Path
Disallow /

noga

Rule Path
Disallow /

nogo

Rule Path
Disallow /

noko

Rule Path
Disallow /

nomad

Rule Path
Disallow /

norb

Rule Path
Disallow /

noxtrumbot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nuse

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nutex

Rule Path
Disallow /

nwsp

Rule Path
Disallow /

obje

Rule Path
Disallow /

ocel

Rule Path
Disallow /

octo

Rule Path
Disallow /

octopus

Rule Path
Disallow /

odi3

Rule Path
Disallow /

oegp

Rule Path
Disallow /

offline

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

offline.explorer

Rule Path
Disallow /

offline.navigator

Rule Path
Disallow /

offline\ explorer

Rule Path
Disallow /

offline\ navigator

Rule Path
Disallow /

ok.mozilla

Rule Path
Disallow /

omg

Rule Path
Disallow /

omni

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

omniexplorer_bot/6.11.1

Rule Path
Disallow /

onfo

Rule Path
Disallow /

onyx

Rule Path
Disallow /

opal

Rule Path
Disallow /

openbot

Rule Path
Disallow /

openf

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openfind data gathere

Rule Path
Disallow /

opentextsitecrawler

Rule Path
Disallow /

openu

Rule Path
Disallow /

orac

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

orbit

Rule Path
Disallow /

oreg

Rule Path
Disallow /

osis

Rule Path
Disallow /

outf

Rule Path
Disallow /

owl

Rule Path
Disallow /

p3p

Rule Path
Disallow /

packrat

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

pagmiedownload

Rule Path
Disallow /

pansci

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

papa

Rule Path
Disallow /

papa\ foto

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

pars

Rule Path
Disallow /

patw

Rule Path
Disallow /

pavu

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

pb2pb

Rule Path
Disallow /

pcbrow

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pear

Rule Path
Disallow /

pecl

Rule Path
Disallow /

pecl::http

Rule Path
Disallow /

peer

Rule Path
Disallow /

peoplepal

Rule Path
Disallow /

pepe

Rule Path
Disallow /

perl

Rule Path
Disallow /

perman

Rule Path
Disallow /

personapilot

Rule Path
Disallow /

persuader

Rule Path
Disallow /

petit

Rule Path
Disallow /

php

Rule Path
Disallow /

php.vers

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

phpot

Rule Path
Disallow /

phras

Rule Path
Disallow /

picalo

Rule Path
Disallow /

piff

Rule Path
Disallow /

pige

Rule Path
Disallow /

pigs

Rule Path
Disallow /

ping

Rule Path
Disallow /

pingalink

Rule Path
Disallow /

pingd

Rule Path
Disallow /

pipe

Rule Path
Disallow /

plag

Rule Path
Disallow /

plant

Rule Path
Disallow /

playstarmusic

Rule Path
Disallow /

pleasecrawl

Rule Path
Disallow /

pluck

Rule Path
Disallow /

pockey

Rule Path
Disallow /

poe\-com

Rule Path
Disallow /

poirot

Rule Path
Disallow /

pomp

Rule Path
Disallow /

port.huron

Rule Path
Disallow /

post

Rule Path
Disallow /

powerset

Rule Path
Disallow /

preload

Rule Path
Disallow /

press

Rule Path
Disallow /

privoxy

Rule Path
Disallow /

probe

Rule Path
Disallow /

program.shareware

Rule Path
Disallow /

progressive.download

Rule Path
Disallow /

propowerbot

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prospector

Rule Path
Disallow /

provider.protocol.discover

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

prowl

Rule Path
Disallow /

proxy

Rule Path
Disallow /

prozilla

Rule Path
Disallow /

psbot

Rule Path
Disallow /

psbot/0.1

Rule Path
Disallow /

psurf

Rule Path
Disallow /

psycheclone

Rule Path
Disallow /

pulse

Rule Path
Disallow /

pump

Rule Path
Disallow /

pushsite

Rule Path
Disallow /

pussycat

Rule Path
Disallow /

puxarapido

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

pyq

Rule Path
Disallow /

pyth

Rule Path
Disallow /

qrva

Rule Path
Disallow /

quepasacreep

Rule Path
Disallow /

query

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

quest

Rule Path
Disallow /

qweer

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

radian

Rule Path
Disallow /

radian6

Rule Path
Disallow /

radiation

Rule Path
Disallow /

rambler

Rule Path
Disallow /

ramp

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reap

Rule Path
Disallow /

recorder

Rule Path
Disallow /

redcarpet

Rule Path
Disallow /

redkernel

Rule Path
Disallow /

reget

Rule Path
Disallow /

relevantnoise

Rule Path
Disallow /

replacer

Rule Path
Disallow /

repo

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

requ

Rule Path
Disallow /

rese

Rule Path
Disallow /

retrieve

Rule Path
Disallow /

rip

Rule Path
Disallow /

rippers\ 0

Rule Path
Disallow /

rix

Rule Path
Disallow /

rma

Rule Path
Disallow /

roboz

Rule Path
Disallow /

rogue

Rule Path
Disallow /

rover

Rule Path
Disallow /

rpt\-http

Rule Path
Disallow /

rsync

Rule Path
Disallow /

rtg30

Rule Path
Disallow /

ruby

Rule Path
Disallow /

rufus

Rule Path
Disallow /

salt

Rule Path
Disallow /

sample

Rule Path
Disallow /

sapo

Rule Path
Disallow /

sauger

Rule Path
Disallow /

savvy

Rule Path
Disallow /

sbider

Rule Path
Disallow /

sbp

Rule Path
Disallow /

scagent

Rule Path
Disallow /

scan

Rule Path
Disallow /

scej\_

Rule Path
Disallow /

sched

Rule Path
Disallow /

schizo

Rule Path
Disallow /

schlong

Rule Path
Disallow /

schmo

Rule Path
Disallow /

scooter

Rule Path
Disallow /

scorp

Rule Path
Disallow /

scout

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

scoutout

Rule Path
Disallow /

scrawl

Rule Path
Disallow /

screen

Rule Path
Disallow /

script

Rule Path
Disallow /

seamonkey$

Rule Path
Disallow /

searchexpress

Rule Path
Disallow /

searchhippo

Rule Path
Disallow /

searchme

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

searchterms

Rule Path
Disallow /

second.street.research

Rule Path
Disallow /

security.kol

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

seeker

Rule Path
Disallow /

sega

Rule Path
Disallow /

semrushbot/0.9

Rule Path
Disallow /

sensis

Rule Path
Disallow /

sept

Rule Path
Disallow /

serious

Rule Path
Disallow /

sezn

Rule Path
Disallow /

seznambot/3.0

Rule Path
Disallow /

shai

Rule Path
Disallow /

share

Rule Path
Disallow /

sharp

Rule Path
Disallow /

shaz

Rule Path
Disallow /

shell

Rule Path
Disallow /

shelo

Rule Path
Disallow /

sherl

Rule Path
Disallow /

shim

Rule Path
Disallow /

shiretoko

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

sicklebot

Rule Path
Disallow /

simple

Rule Path
Disallow /

siph

Rule Path
Disallow /

site.sniper

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

sitecheck

Rule Path
Disallow /

sitecheck\.internetseer\.com

Rule Path
Disallow /

sitecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

sitevigil

Rule Path
Disallow /

sitex

Rule Path
Disallow /

sleip

Rule Path
Disallow /

slide

Rule Path
Disallow /

slurpy.verifier

Rule Path
Disallow /

sly

Rule Path
Disallow /

smag

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

smurf

Rule Path
Disallow /

sna\-

Rule Path
Disallow /

snag

Rule Path
Disallow /

snake

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

snip

Rule Path
Disallow /

snoop

Rule Path
Disallow /

snoopy

Rule Path
Disallow /

so\-net

Rule Path
Disallow /

socialspider-finder/0.2

Rule Path
Disallow /

socsci

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogouwebspider/3.0

Rule Path
Disallow /

sogouwebspider/4.0

Rule Path
Disallow /

sohu

Rule Path
Disallow /

solr

Rule Path
Disallow /

sootle

Rule Path
Disallow /

soso

Rule Path
Disallow /

sosospider+

Rule Path
Disallow /

spacebison

Rule Path
Disallow /

spad

Rule Path
Disallow /

span

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

speed

Rule Path
Disallow /

spegla

Rule Path
Disallow /

sphere

Rule Path
Disallow /

sphider

Rule Path
Disallow /

spider

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

spiderengine

Rule Path
Disallow /

spiderview

Rule Path
Disallow /

spin

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

sproose

Rule Path
Disallow /

spurl

Rule Path
Disallow /

spyder

Rule Path
Disallow /

sq.webscanner

Rule Path
Disallow /

squi

Rule Path
Disallow /

sqwid

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

ssm\_ag

Rule Path
Disallow /

stack

Rule Path
Disallow /

stamina

Rule Path
Disallow /

stamp

Rule Path
Disallow /

stanford

Rule Path
Disallow /

statbot

Rule Path
Disallow /

state

Rule Path
Disallow /

steel

Rule Path
Disallow /

steeler

Rule Path
Disallow /

strateg

Rule Path
Disallow /

stress

Rule Path
Disallow /

strip

Rule Path
Disallow /

studybot

Rule Path
Disallow /

style

Rule Path
Disallow /

subot

Rule Path
Disallow /

suck

Rule Path
Disallow /

sume

Rule Path
Disallow /

sun4m

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superbro

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

supervi

Rule Path
Disallow /

surf4me

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surfwalker

Rule Path
Disallow /

susi

Rule Path
Disallow /

suza

Rule Path
Disallow /

suzu

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

sweep

Rule Path
Disallow /

sygol

Rule Path
Disallow /

syncrisis

Rule Path
Disallow /

systems

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

tagger

Rule Path
Disallow /

tagyu

Rule Path
Disallow /

take

Rule Path
Disallow /

takeout

Rule Path
Disallow /

talkro

Rule Path
Disallow /

talwinhttpclient

Rule Path
Disallow /

tamu

Rule Path
Disallow /

tandem

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

tarspider

Rule Path
Disallow /

tbot

Rule Path
Disallow /

tcf

Rule Path
Disallow /

tcs\/1

Rule Path
Disallow /

teamsoft

Rule Path
Disallow /

tecomi

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleport\ pro

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

templeton

Rule Path
Disallow /

tencent

Rule Path
Disallow /

terrawiz

Rule Path
Disallow /

test

Rule Path
Disallow /

texnut

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

the.intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

thomas

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

timely

Rule Path
Disallow /

titan

Rule Path
Disallow /

tmcrawler

Rule Path
Disallow /

tmhtload

Rule Path
Disallow /

toata\ dragostea\ mea\ pentru\ diavola

Rule Path
Disallow /

tocrawl

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

todobr

Rule Path
Disallow /

tongco

Rule Path
Disallow /

topic

Rule Path
Disallow /

torrent

Rule Path
Disallow /

track

Rule Path
Disallow /

translate

Rule Path
Disallow /

traveler

Rule Path
Disallow /

treeview

Rule Path
Disallow /

trivial

Rule Path
Disallow /

true

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

true_robot/1.0

Rule Path
Disallow /

tunnel

Rule Path
Disallow /

turing

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

tutorgig

Rule Path
Disallow /

tv33\_mercator

Rule Path
Disallow /

twat

Rule Path
Disallow /

tweak

Rule Path
Disallow /

twice

Rule Path
Disallow /

twisted.pagegetter

Rule Path
Disallow /

tygo

Rule Path
Disallow /

ubee

Rule Path
Disallow /

ucmore

Rule Path
Disallow /

udmsearch

Rule Path
Disallow /

uiowacrawler

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

umbc

Rule Path
Disallow /

unf

Rule Path
Disallow /

universalfeedparser

Rule Path
Disallow /

unknown

Rule Path
Disallow /

upg1

Rule Path
Disallow /

uri::fetch

Rule Path
Disallow /

url.control

Rule Path
Disallow /

url\_spider\_pro

Rule Path
Disallow /

urlbase

Rule Path
Disallow /

urldispatcher

Rule Path
Disallow /

urlgetfile

Rule Path
Disallow /

urllib

Rule Path
Disallow /

urlspiderpro

Rule Path
Disallow /

urly

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

user\-agent

Rule Path
Disallow /

useragent

Rule Path
Disallow /

user-agent

Rule Path
Disallow /

usyd

Rule Path
Disallow /

utilmind

Rule Path
Disallow /

vacuum

Rule Path
Disallow /

vagabo

Rule Path
Disallow /

valet

Rule Path
Disallow /

valid

Rule Path
Disallow /

vamp

Rule Path
Disallow /

vayala

Rule Path
Disallow /

vb\_

Rule Path
Disallow /

vci

Rule Path
Disallow /

vci webviewer vci webviewer win32

Rule Path
Disallow /

veri\~li

Rule Path
Disallow /

verif

Rule Path
Disallow /

versus

Rule Path
Disallow /

via

Rule Path
Disallow /

viewer

Rule Path
Disallow /

virtual

Rule Path
Disallow /

visibilitygap

Rule Path
Disallow /

visual

Rule Path
Disallow /

vobsub

Rule Path
Disallow /

void

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

vspider

Rule Path
Disallow /

vsyn

Rule Path
Disallow /

w\:pacbho60

Rule Path
Disallow /

w0000t

Rule Path
Disallow /

w3c

Rule Path
Disallow /

w3m

Rule Path
Disallow /

w3search

Rule Path
Disallow /

walhello

Rule Path
Disallow /

walker

Rule Path
Disallow /

wand

Rule Path
Disallow /

waol

Rule Path
Disallow /

wapt

Rule Path
Disallow /

watch

Rule Path
Disallow /

wavefire

Rule Path
Disallow /

wbdbot

Rule Path
Disallow /

weather

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

web.by.mail

Rule Path
Disallow /

web.data.extractor

Rule Path
Disallow /

web.downloader

Rule Path
Disallow /

web.ima

Rule Path
Disallow /

web.mole

Rule Path
Disallow /

web.sucker

Rule Path
Disallow /

web\ image\ collector

Rule Path
Disallow /

web\ sucker

Rule Path
Disallow /

web2mal

Rule Path
Disallow /

web2wap

Rule Path
Disallow /

webalta

Rule Path
Disallow /

webaltbot

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webbandit/3.50

Rule Path
Disallow /

webbot

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

webcat

Rule Path
Disallow /

webclip

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcollector

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webcor

Rule Path
Disallow /

webcraft\@bea

Rule Path
Disallow /

webcrawl

Rule Path
Disallow /

webdat

Rule Path
Disallow /

webdav

Rule Path
Disallow /

webdevil

Rule Path
Disallow /

webdownloader

Rule Path
Disallow /

webdup

Rule Path
Disallow /

webemail

Rule Path
Disallow /

webemailextrac

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webfilter

Rule Path
Disallow /

webfountain

Rule Path
Disallow /

webgo

Rule Path
Disallow /

webgo\ is

Rule Path
Disallow /

webhook

Rule Path
Disallow /

webinator

Rule Path
Disallow /

webind

Rule Path
Disallow /

webitpr

Rule Path
Disallow /

weblea

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webmaster

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webmin

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webmole

Rule Path
Disallow /

webpic

Rule Path
Disallow /

webpin

Rule Path
Disallow /

webpix

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webripper

Rule Path
Disallow /

webrobot

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website

Rule Path
Disallow /

website quester

Rule Path
Disallow /

website.extractor

Rule Path
Disallow /

website.quester

Rule Path
Disallow /

website\ extractor

Rule Path
Disallow /

website\ quester

Rule Path
Disallow /

websnake

Rule Path
Disallow /

webspider

Rule Path
Disallow /

webster

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

websucker

Rule Path
Disallow /

webtre

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webwasher

Rule Path
Disallow /

webweasel

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

wells

Rule Path
Disallow /

wells\ search\ ii

Rule Path
Disallow /

wep.search.00

Rule Path
Disallow /

wep\ search

Rule Path
Disallow /

wep\_s

Rule Path
Disallow /

werelatebot

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget/1.5.3

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

whack

Rule Path
Disallow /

whacker

Rule Path
Disallow /

whiz

Rule Path
Disallow /

whostalking

Rule Path
Disallow /

widow

Rule Path
Disallow /

wildsoft.surfer

Rule Path
Disallow /

win67

Rule Path
Disallow /

window.location

Rule Path
Disallow /

windows.95\)

Rule Path
Disallow /

windows.95\;

Rule Path
Disallow /

windows.98\)

Rule Path
Disallow /

windows.98\;

Rule Path
Disallow /

winht

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

winhttprequest

Rule Path
Disallow /

winhttrack

Rule Path
Disallow /

winnie.poh

Rule Path
Disallow /

winodws

Rule Path
Disallow /

wire

Rule Path
Disallow /

wisebot

Rule Path
Disallow /

wisenutbot

Rule Path
Disallow /

wish

Rule Path
Disallow /

wizz

Rule Path
Disallow /

wordp

Rule Path
Disallow /

works

Rule Path
Disallow /

world

Rule Path
Disallow /

wumpus

Rule Path
Disallow /

wweb

Rule Path
Disallow /

www.mechanize

Rule Path
Disallow /

www.ranks.nl

Rule Path
Disallow /

www\-collector

Rule Path
Disallow /

wwwc

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

wwwster

Rule Path
Disallow /

x\-tractor

Rule Path
Disallow /

x12r1

Rule Path
Disallow /

xaldon

Rule Path
Disallow /

xaldon\ webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xget

Rule Path
Disallow /

xirq

Rule Path
Disallow /

y\!oasis

Rule Path
Disallow /

y\!tunnel

Rule Path
Disallow /

yacy

Rule Path
Disallow /

yadirectbot

Rule Path
Disallow /

yahoo\-mmaudvid

Rule Path
Disallow /

yahooseeker

Rule Path
Disallow /

yahooysmcm

Rule Path
Disallow /

yamm

Rule Path
Disallow /

yand

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yang

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yoono

Rule Path
Disallow /

yori

Rule Path
Disallow /

yotta

Rule Path
Disallow /

ytunnel

Rule Path
Disallow /

zade

Rule Path
Disallow /

zagre

Rule Path
Disallow /

zbot

Rule Path
Disallow /

zeal

Rule Path
Disallow /

zebot

Rule Path
Disallow /

zermelo

Rule Path
Disallow /

zerx

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zeus\.*webster

Rule Path
Disallow /

zipcode

Rule Path
Disallow /

zixy

Rule Path
Disallow /

zmao

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top