Log In Sign Up

kotly.com
robots.txt

Robots Exclusion Standard data for kotly.com

Resource Scan

Scan Details

Site Domain kotly.com
Base Domain kotly.com
Scan Status Ok
Last Scan2024-11-07T02:03:11+00:00
Next Scan 2024-12-07T02:03:11+00:00

Last Scan

Scanned2024-11-07T02:03:11+00:00
URL https://kotly.com/robots.txt
Domain IPs 31.186.83.220
Response IP 31.186.83.220
Found Yes
Hash df526db08564aba8e14ddd4e0ed514baee59cdd83e24af69be422cca76c9549b
SimHash 7b828673688f

Groups

*

Rule Path
Disallow /pl/*.html?search_query=*
Disallow /de/*.html?search_query=*
Disallow /gb/*.html?search_query=*
Disallow /cs/*.html?search_query=*
Disallow /pl/module/wtblocksearch/catesearch?*
Disallow /de/module/wtblocksearch/catesearch?*
Disallow /gb/module/wtblocksearch/catesearch?*
Disallow /cs/module/wtblocksearch/catesearch?*
Disallow /popup_add_image.php*
Disallow /advanced_search_result.php*
Disallow /tab.php*
Disallow /information.php*

a1 sitemap generator

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

abachobot

Rule Path
Disallow /

abby

Rule Path
Disallow /

abcdatos botlink

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

abrave spider

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

accoona-ai-agent

Rule Path
Disallow /

acme.spider

Rule Path
Disallow /

acme\.spider

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

addsugarspiderbot

Rule Path
Disallow /

addthis.com

Rule Path
Disallow /

adidxbot

Rule Path
Disallow /

admantx platform semantic analyzer

Rule Path
Disallow /

adressendeutschland.de

Rule Path
Disallow /

ahoy! the homepage finder

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aitcsrobot/1.1

Rule Path
Disallow /

akula

Rule Path
Disallow /

alexa

Rule Path
Disallow /

alexa site audit

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

alexandria(\s|\+)prototype(\s|\+)project

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alkaline

Rule Path
Disallow /

allentrack

Rule Path
Disallow /

almaden

Rule Path
Disallow /

amagit.com

Rule Path
Disallow /

amibot

Rule Path
Disallow /

amznkassocbot

Rule Path
Disallow /

ananzi

Rule Path
Disallow /

antbot

Rule Path
Disallow /

anthill

Rule Path
Disallow /

apercite

Rule Path
Disallow /

aportworm

Rule Path
Disallow /

appie

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

arabot

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

arachnode.net

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

arale

Rule Path
Disallow /

araneo

Rule Path
Disallow /

araybot

Rule Path
Disallow /

architext

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archive\.org_bot

Rule Path
Disallow /

aretha

Rule Path
Disallow /

ariadne

Rule Path
Disallow /

arks

Rule Path
Disallow /

ask jeeves/teoma

Rule Path
Disallow /

askjeeves

Rule Path
Disallow /

aspider (associative spider)

Rule Path
Disallow /

aspider/0.09

Rule Path
Disallow /

asterias

Rule Path
Disallow /

atn worldwide

Rule Path
Disallow /

atomz

Rule Path
Disallow /

atomz.com search robot

Rule Path
Disallow /

auresys

Rule Path
Disallow /

auresys/1.0

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

automattic analytics crawler

Rule Path
Disallow /

awbot

Rule Path
Disallow /

b2w/0.1

Rule Path
Disallow /

babaloospider

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backlink-check.de

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

backrub

Rule Path
Disallow /

backrub/.

Rule Path
Disallow /

bad-neighborhood

Rule Path
Disallow /

bay spider

Rule Path
Disallow /

baypup

Rule Path
Disallow /

bbot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

begunadvertising

Rule Path
Disallow /

beslistbot

Rule Path
Disallow /

biadu

Rule Path
Disallow /

big brother

Rule Path
Disallow /

biglotron

Rule Path
Disallow /

bitlybot

Rule Path
Disallow /

biwec

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

bizbot003

Rule Path
Disallow /

bizbot04 kirk.overleaf.com

Rule Path
Disallow /

bjaaland

Rule Path
Disallow /

bl.uk_lddc_bot

Rule Path
Disallow /

black hole

Rule Path
Disallow /

black.hole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blaiz\-bee

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blinkacrawler

Rule Path
Disallow /

bloglines

Rule Path
Disallow /

blogpulse

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

bnf.fr_bot

Rule Path
Disallow /

boitho\.com\-dc

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

bookmark-tool

Rule Path
Disallow /

bookmark\-manager

Rule Path
Disallow /

borg-bot

Rule Path
Disallow /

bot

Rule Path
Disallow /

bot mailto: craftbot@yahoo.com

Rule Path
Disallow /

bot mailto:craftbot@yahoo.com

Rule Path
Disallow /

bot-pge.chlooe.com

Rule Path
Disallow /

bot.wsowner.com

Rule Path
Disallow /

botalot

Rule Path
Disallow /

botmobi

Rule Path
Disallow /

botonparade

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

boxseabot

Rule Path
Disallow /

brainbrubot

Rule Path
Disallow /

bright.net caching robot

Rule Path
Disallow /

browsershots

Rule Path
Disallow /

brutus\/aet

Rule Path
Disallow /

bspider

Rule Path
Disallow /

bspider/1.0 libwww-perl/0.40

Rule Path
Disallow /

bubing

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

bwh3_user_agent

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

calif

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

cassandra

Rule Path
Disallow /

castabot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ccresearchbot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

celestial

Rule Path
Disallow /

cfnetwork|checkbot

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

checkprivacy

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

china\slocal\sbrowse\s2\.6

Rule Path
Disallow /

christcrawler.com

Rule Path
Disallow /

churl

Rule Path
Disallow /

cienciaficcion.net

Rule Path
Disallow /

cirrusexplorer

Rule Path
Disallow /

city review

Rule Path
Disallow /

cityreview

Rule Path
Disallow /

classbot

Rule Path
Disallow /

cligoorobot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cloakdetect

Rule Path
Disallow /

cloudflare-alwaysonline

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

cmc/0.01

Rule Path
Disallow /

cms crawler

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

code\ssample\sweb\sclient

Rule Path
Disallow /

collective

Rule Path
Disallow /

combine

Rule Path
Disallow /

combine system

Rule Path
Disallow /

commons\-httpclient

Rule Path
Disallow /

company news search engine

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

computingsite robi/1.0

Rule Path
Disallow /

conceptbot

Rule Path
Disallow /

confuzzledbot

Rule Path
Disallow /

contentmatch

Rule Path
Disallow /

contentsmartz

Rule Path
Disallow /

coolbot

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copyright sheriff

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

core

Rule Path
Disallow /

corpuscrawler

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

covario-ids

Rule Path
Disallow /

coverscout

Rule Path
Disallow /

crawl

Rule Path
Disallow /

crawler

Rule Path
Disallow /

crawler for netopian

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

crowsnest

Rule Path
Disallow /

curious george

Rule Path
Disallow /

cursor

Rule Path
Disallow /

cusco

Rule Path
Disallow /

custo

Rule Path
Disallow /

cyberpatrol sitecat webbot

Rule Path
Disallow /

cyberspyder link test

Rule Path
Disallow /

cydralspider

Rule Path
Disallow /

datacha0s\/2\.0

Rule Path
Disallow /

dataprovider.com

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

dblbot

Rule Path
Disallow /

dcpbot

Rule Path
Disallow /

dealgates bot

Rule Path
Disallow /

demo\sbot

Rule Path
Disallow /

desert realm spider

Rule Path
Disallow /

deweb(c) katalog/index

Rule Path
Disallow /

die blinde kuh

Rule Path
Disallow /

die intraformant

Rule Path
Disallow /

dienstspider

Rule Path
Disallow /

digger

Rule Path
Disallow /

digimarc marcspider

Rule Path
Disallow /

digimarc marcspider/cgi

Rule Path
Disallow /

digital integrity robot

Rule Path
Disallow /

dinoping

Rule Path
Disallow /

direct hit grabber

Rule Path
Disallow /

disco

Rule Path
Disallow /

disco pump 3.0

Rule Path
Disallow /

disco pump 3.2

Rule Path
Disallow /

disco pump 3/2

Rule Path
Disallow /

disco pumpe 3,0

Rule Path
Disallow /

discofinder

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dkimrepbot

Rule Path
Disallow /

dlvr.it/1.0

Rule Path
Disallow /

dnabot

Rule Path
Disallow /

dns-digger-explorer

Rule Path
Disallow /

docomo

Rule Path
Disallow /

domaindb

Rule Path
Disallow /

dot tk – spider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotnetdotcom

Rule Path
Disallow /

dotsemantic

Rule Path
Disallow /

download demon

Rule Path
Disallow /

download demon/3.2.0.8

Rule Path
Disallow /

download demon/3.5.0.11

Rule Path
Disallow /

download express

Rule Path
Disallow /

download\+master

Rule Path
Disallow /

dragonbot

Rule Path
Disallow /

dripfeedbot

Rule Path
Disallow /

drupact

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

dtsearchspider

Rule Path
Disallow /

dumbot

Rule Path
Disallow /

dwcp (dridus’ web cataloging project)

Rule Path
Disallow /

e-collector

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

easydl

Rule Path
Disallow /

ebiness

Rule Path
Disallow /

ecairn-grabber

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

ecatch/3.0

Rule Path
Disallow /

ecommercebot

Rule Path
Disallow /

edisterbot

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

eit link verifier robot

Rule Path
Disallow /

elfinbot

Rule Path
Disallow /

emacs-w3 search engine

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

embedly

Rule Path
Disallow /

emefgebot

Rule Path
Disallow /

envolk

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

es

Rule Path
Disallow /

esculapio

Rule Path
Disallow /

esribot

Rule Path
Disallow /

esther

Rule Path
Disallow /

euripbot

Rule Path
Disallow /

eurobot

Rule Path
Disallow /

eventgurubot

Rule Path
Disallow /

everyonesocialbot

Rule Path
Disallow /

evliya celebi

Rule Path
Disallow /

evrinid

Rule Path
Disallow /

exabot

Rule Path
Disallow /

exb language crawler

Rule Path
Disallow /

express webpictures

Rule Path
Disallow /

express webpictures (www.express-soft.com)

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

facebookplatform

Rule Path
Disallow /

factbot

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

fairad kunde

Rule Path
Disallow /

fairshare

Rule Path
Disallow /

falconsbot

Rule Path
Disallow /

fast-webcrawler

Rule Path
Disallow /

fastbot crawler

Rule Path
Disallow /

fastcrawler

Rule Path
Disallow /

faubot

Rule Path
Disallow /

favorg

Rule Path
Disallow /

fdm(\s|\+)1

Rule Path
Disallow /

fdse robot

Rule Path
Disallow /

feedburner

Rule Path
Disallow /

feedcatbot

Rule Path
Disallow /

feedfetcher

Rule Path
Disallow /

feedfinder/bloggz.se

Rule Path
Disallow /

feedly

Rule Path
Disallow /

felix ide

Rule Path
Disallow /

ferret

Rule Path
Disallow /

fetch(\s|\+)api(\s|\+)request

Rule Path
Disallow /

fetch-guess

Rule Path
Disallow /

fetchrover

Rule Path
Disallow /

fido

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

firmilybot

Rule Path
Disallow /

fish search

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flashget webwasher 3.2

Rule Path
Disallow /

flashget webwasher 3/2

Rule Path
Disallow /

flatland industries web spider

Rule Path
Disallow /

flightdeckreportsbot

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

flocke bot

Rule Path
Disallow /

fluid dynamics search engine robot

Rule Path
Disallow /

followsite bot

Rule Path
Disallow /

foobot

Rule Path
Disallow /

fooooo_web_video_crawl

Rule Path
Disallow /

fouineur

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

frontpage [nc, or]

Rule Path
Disallow /

frontpage [nc,or]

Rule Path
Disallow /

fulltext

Rule Path
Disallow /

funnelback

Rule Path
Disallow /

funnelweb

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

gammaspider, focusedcrawler

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

gazz

Rule Path
Disallow /

gcreep

Rule Path
Disallow /

geliyoobot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

genieo web filter

Rule Path
Disallow /

getbot

Rule Path
Disallow /

getright

Rule Path
Disallow /

getright/2.11

Rule Path
Disallow /

getright/3.1

Rule Path
Disallow /

getright/3.2

Rule Path
Disallow /

getright/3.3

Rule Path
Disallow /

getright/3.3.3

Rule Path
Disallow /

getright/3.3.4

Rule Path
Disallow /

getright/4.0.0

Rule Path
Disallow /

getright/4.1.0

Rule Path
Disallow /

getright/4.1.1

Rule Path
Disallow /

getright/4.1.2

Rule Path
Disallow /

getright/4.2

Rule Path
Disallow /

getright/4.2b (portuguxeas)

Rule Path
Disallow /

getright/4.2c

Rule Path
Disallow /

getright/4.3

Rule Path
Disallow /

getright/4.5

Rule Path
Disallow /

getright/4.5a

Rule Path
Disallow /

getright/4.5b

Rule Path
Disallow /

getright/4.5b1

Rule Path
Disallow /

getright/4.5b2

Rule Path
Disallow /

getright/4.5b3

Rule Path
Disallow /

getright/4.5b6

Rule Path
Disallow /

getright/4.5b7

Rule Path
Disallow /

getright/4.5c

Rule Path
Disallow /

getright/4.5d

Rule Path
Disallow /

getright/4.5e

Rule Path
Disallow /

getright/5.0beta1

Rule Path
Disallow /

getright/5.0beta2

Rule Path
Disallow /

getterroboplus puu

Rule Path
Disallow /

geturl

Rule Path
Disallow /

geturl.rexx v1.05

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

gingercrawler

Rule Path
Disallow /

girafabot

Rule Path
Disallow /

gnodspider

Rule Path
Disallow /

go! zilla

Rule Path
Disallow /

go! zilla (www.gozilla.com)

Rule Path
Disallow /

go! zilla 3.3 (www.gozilla.com)

Rule Path
Disallow /

go! zilla 3.5 (www.gozilla.com)

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go!zilla (www.gozilla.com)

Rule Path
Disallow /

go!zilla 3.3 (www.gozilla.com)

Rule Path
Disallow /

go!zilla 3.5 (www.gozilla.com)

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

gocrawl

Rule Path
Disallow /

goforitbot

Rule Path
Disallow /

goldfire(\s|\+)server

Rule Path
Disallow /

golem

Rule Path
Disallow /

golem/1.1

Rule Path
Disallow /

gonzo

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

grahambot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

grapnel/0.01 experiment

Rule Path
Disallow /

griffon

Rule Path
Disallow /

gromit

Rule Path
Disallow /

gromit/1.0

Rule Path
Disallow /

grub

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

gulliver

Rule Path
Disallow /

gulper bot

Rule Path
Disallow /

gurujibot

Rule Path
Disallow /

hãƒâ¤mãƒâ¤hãƒâ¤kki

Rule Path
Disallow /

hailoobot

Rule Path
Disallow /

hambot

Rule Path
Disallow /

happyfunbot

Rule Path
Disallow /

harvest

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

hatena antenna

Rule Path
Disallow /

hatenascreenshot

Rule Path
Disallow /

havindex

Rule Path
Disallow /

hawkreader

Rule Path
Disallow /

heartrails_capture

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

hi (html index) search

Rule Path
Disallow /

hku www octopus

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hl_ftien_spider

Rule Path
Disallow /

hmview

Rule Path
Disallow /

holmes

Rule Path
Disallow /

holmesbot

Rule Path
Disallow /

hometags

Rule Path
Disallow /

hometown spider pro

Rule Path
Disallow /

hosttracker

Rule Path
Disallow /

hosttracker.com

Rule Path
Disallow /

ht://dig

Rule Path
Disallow /

htdig

Rule Path
Disallow /

htmlgobble

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

html_analyzer

Rule Path
Disallow /

httpcomponents\/1.1

Rule Path
Disallow /

httpfetcher

Rule Path
Disallow /

httpget\-5\.2\.2

Rule Path
Disallow /

httpget\?5\.2\.2

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack

Rule Path
Disallow /

httrack 3,0

Rule Path
Disallow /

httrack 3.0

Rule Path
Disallow /

httrack [nc, or]

Rule Path
Disallow /

httrack [nc,or]

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

hubspot connect

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

hyper-decontextualizer

Rule Path
Disallow /

i, robot

Rule Path
Disallow /

iajabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ibm_planetwide

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

icjobs

Rule Path
Disallow /

ideelaborplagiaat

Rule Path
Disallow /

iframely

Rule Path
Disallow /

iktomi

Rule Path
Disallow /

ilse

Rule Path
Disallow /

image stripper

Rule Path
Disallow /

image sucker

Rule Path
Disallow /

image.kapsi.net

Rule Path
Disallow /

imagelock

Rule Path
Disallow /

imbot

Rule Path
Disallow /

immediatenet thumbnails

Rule Path
Disallow /

inagist.com url crawler

Rule Path
Disallow /

incywincy

Rule Path
Disallow /

indy library

Rule Path
Disallow /

indy library [nc, or]

Rule Path
Disallow /

indy library [nc,or]

Rule Path
Disallow /

indy-bibliothek

Rule Path
Disallow /

influencebot

Rule Path
Disallow /

infohelfer

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

informant

Rule Path
Disallow /

infoseek robot 1.0

Rule Path
Disallow /

infoseek sidewinder

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

ingrid

Rule Path
Disallow /

inktomi slurp

Rule Path
Disallow /

inspector web

Rule Path
Disallow /

integromedb

Rule Path
Disallow /

intelliagent

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet cruiser robot

Rule Path
Disallow /

internet ninja

Rule Path
Disallow /

internet ninja 4,0

Rule Path
Disallow /

internet ninja 4.0

Rule Path
Disallow /

internet ninja 5,0

Rule Path
Disallow /

internet ninja 5.0

Rule Path
Disallow /

internet ninja 6,0

Rule Path
Disallow /

internet ninja 6.0

Rule Path
Disallow /

internet shinchakubin

Rule Path
Disallow /

internetseer

Rule Path
Disallow /

intute

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

iron33

Rule Path
Disallow /

iron33/1.0.2

Rule Path
Disallow /

isilox

Rule Path
Disallow /

israeli-search

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

ixebot

Rule Path
Disallow /

jabse.com crawler

Rule Path
Disallow /

jadynavebot

Rule Path
Disallow /

jakarta\+commons\-httpclient

Rule Path
Disallow /

java

Rule Path
Disallow /

javabee

Rule Path
Disallow /

jbot java web robot

Rule Path
Disallow /

jcrawler

Rule Path
Disallow /

jeeves

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

job roboter spider

Rule Path
Disallow /

jobo

Rule Path
Disallow /

jobo java web robot

Rule Path
Disallow /

jobot

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

joebot

Rule Path
Disallow /

jumpstation

Rule Path
Disallow /

just-crawler

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

kalooga

Rule Path
Disallow /

karneval-bot

Rule Path
Disallow /

katipo

Rule Path
Disallow /

kdd-explorer

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

kenjin.spider

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

keyword.density

Rule Path
Disallow /

keyworddensityrobot

Rule Path
Disallow /

kilroy

Rule Path
Disallow /

kit-fireball

Rule Path
Disallow /

komodiabot

Rule Path
Disallow /

ko_yappo_robot

Rule Path
Disallow /

kraken

Rule Path
Disallow /

krowler

Rule Path
Disallow /

kulturarw

Rule Path
Disallow /

kyluka

Rule Path
Disallow /

l.webis

Rule Path
Disallow /

labelgrabber

Rule Path
Disallow /

larbin

Rule Path
Disallow /

larbin (samualt9@bigfoot.com)

Rule Path
Disallow /

larbin samualt9@bigfoot.com

Rule Path
Disallow /

larbin_2.6.2 (kabura@sushi.com)

Rule Path
Disallow /

larbin_2.6.2 (larbin2.6.2 @ unspecified.mail)

Rule Path
Disallow /

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

Rule Path
Disallow /

larbin_2.6.2 (listonatccdotgatechdotedu)

Rule Path
Disallow /

larbin_2.6.2 (vitalbox1@hotmail.com)

Rule Path
Disallow /

larbin_2.6.2 kabura@sushi.com

Rule Path
Disallow /

larbin_2.6.2 larbin2.6.2 @ unspecified.mail

Rule Path
Disallow /

larbin_2.6.2 larbin2.6.2@unspecified.mail

Rule Path
Disallow /

larbin_2.6.2 larbin@correa.org

Rule Path
Disallow /

larbin_2.6.2 listonatccdotgatechdotedu

Rule Path
Disallow /

larbin_2.6.2 vitalbox1@hotmail.com

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

legs

Rule Path
Disallow /

leikibot

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

libweb / clshttp

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

libwww

Rule Path
Disallow /

libwww\-perl

Rule Path
Disallow /

lijit

Rule Path
Disallow /

lilina

Rule Path
Disallow /

linguabot

Rule Path
Disallow /

linguatools

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

link valet online

Rule Path
Disallow /

link validator

Rule Path
Disallow /

linkaider

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

linkchecker.sourceforge.net

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linklint-checkonly

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkscan/8.1a unix

Rule Path
Disallow /

linkscan/8.1a.unix

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

lipperhey spider

Rule Path
Disallow /

livedoor screenshot

Rule Path
Disallow /

livejournal\.com

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

loadimpactpageanalyzer

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

lockon

Rule Path
Disallow /

lockss

Rule Path
Disallow /

logo.gif crawler

Rule Path
Disallow /

luminatebot

Rule Path
Disallow /

lwp

Rule Path
Disallow /

lwp-request

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

lwp-trivial/1.34

Rule Path
Disallow /

lwp-triviale

Rule Path
Disallow /

lwp\-request

Rule Path
Disallow /

lwp\-tivial

Rule Path
Disallow /

lwp\-trivial

Rule Path
Disallow /

lwp\:\:simple

Rule Path
Disallow /

lycos

Rule Path
Disallow /

lycos[_+]

Rule Path
Disallow /

mac wwwworm

Rule Path
Disallow /

magpie

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru bot

Rule Path
Disallow /

marcedit.5.2.web.client

Rule Path
Disallow /

marvin/infoseek

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

mass downloader/2.2

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mata.hari

Rule Path
Disallow /

mattie

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mediafox

Rule Path
Disallow /

megite

Rule Path
Disallow /

memonewsbot

Rule Path
Disallow /

merzscope

Rule Path
Disallow /

metageneratorcrawler

Rule Path
Disallow /

metaheadersbot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

metamojicrawler

Rule Path
Disallow /

metaspinner/0.01

Rule Path
Disallow /

metauri

Rule Path
Disallow /

metauri api

Rule Path
Disallow /

mia bot

Rule Path
Disallow /

miadev

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

microsoft url control – 5.01.4511

Rule Path
Disallow /

microsoft url control – 6.00.8169

Rule Path
Disallow /

microsoft url control - 5.01.4511

Rule Path
Disallow /

microsoft url control - 6.00.8169

Rule Path
Disallow /

microsoft(\s|\+)url(\s|\+)control

Rule Path
Disallow /

microsoft.url

Rule Path
Disallow /

midown tool

Rule Path
Disallow /

midown werkzeug

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

miixpc/4.2

Rule Path
Disallow /

milbot

Rule Path
Disallow /

mimas

Rule Path
Disallow /

mindcrawler

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mister pix ii 2.01

Rule Path
Disallow /

mister pix ii 2.02a

Rule Path
Disallow /

mister pix ii 2/01

Rule Path
Disallow /

mister pix version.dll

Rule Path
Disallow /

mister.pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

mnogosearch

Rule Path
Disallow /

mnogosearch search engine software

Rule Path
Disallow /

moba-crawler

Rule Path
Disallow /

moget

Rule Path
Disallow /

moget/2.1

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

momspider

Rule Path
Disallow /

monster

Rule Path
Disallow /

morfeus

Rule Path
Disallow /

motor

Rule Path
Disallow /

motoricerca-robots.txt-checker

Rule Path
Disallow /

mp3bot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msr-isrccrawler

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

muncher

Rule Path
Disallow /

muninn

Rule Path
Disallow /

muscat ferret

Rule Path
Disallow /

muscatferre

Rule Path
Disallow /

musobot

Rule Path
Disallow /

mwd.search

Rule Path
Disallow /

myweb

Rule Path
Disallow /

nabot

Rule Path
Disallow /

nagios

Rule Path
Disallow /

najdi.si

Rule Path
Disallow /

nalezenczbot

Rule Path
Disallow /

naver

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

navroad

Rule Path
Disallow /

ndspider

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

nec-meshexplorer

Rule Path
Disallow /

nederland.zoek

Rule Path
Disallow /

nekstbot

Rule Path
Disallow /

neosciocrawler

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

net vampire

Rule Path
Disallow /

net vampire/3.0

Rule Path
Disallow /

netants

Rule Path
Disallow /

netants/1.10

Rule Path
Disallow /

netants/1.23

Rule Path
Disallow /

netants/1.24

Rule Path
Disallow /

netants/1.25

Rule Path
Disallow /

netcarta cyberpilot pro

Rule Path
Disallow /

netcarta webmap engine

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

netestate crawler

Rule Path
Disallow /

netluchs

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

netscoop

Rule Path
Disallow /

netseer

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netzip

Rule Path
Disallow /

netzip downloader 1.0 win32 (12. november 1998)

Rule Path
Disallow /

netzip downloader 1.0 win32(nov 12 1998)

Rule Path
Disallow /

netzip-downloader/1.0.62 (win32; 7. dezember 1998)

Rule Path
Disallow /

netzip-downloader/1.0.62 (win32; dec 7 1998)

Rule Path
Disallow /

netzippy + (http://www.innerprise.net/usp-spider.asp)

Rule Path
Disallow /

netzippy+(http://www.innerprise.net/usp-spider.asp)

Rule Path
Disallow /

newscan-online

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

ng\/2\.

Rule Path
Disallow /

nhse web forager

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nigma.ru

Rule Path
Disallow /

ning

Rule Path
Disallow /

nlnz_iaharvester2013

Rule Path
Disallow /

nodestackbot

Rule Path
Disallow /

nomad

Rule Path
Disallow /

northern light gulliver

Rule Path
Disallow /

no_user_agent

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nuhk

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nworm

Rule Path
Disallow /

nymesis

Rule Path
Disallow /

nzexplorer

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

obot

Rule Path
Disallow /

occam

Rule Path
Disallow /

ocelli

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

offline explorer/1.2

Rule Path
Disallow /

offline explorer/1.4

Rule Path
Disallow /

offline explorer/1.6

Rule Path
Disallow /

offline explorer/1.7

Rule Path
Disallow /

offline explorer/1.9

Rule Path
Disallow /

offline explorer/2.0

Rule Path
Disallow /

offline explorer/2.1

Rule Path
Disallow /

offline explorer/2.3

Rule Path
Disallow /

offline explorer/2.4

Rule Path
Disallow /

offline explorer/2.5

Rule Path
Disallow /

offline navigator

Rule Path
Disallow /

offline(\s|\+)navigator

Rule Path
Disallow /

offline-navigator

Rule Path
Disallow /

offline.explorer

Rule Path
Disallow /

ogspider

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

onetszukaj

Rule Path
Disallow /

ontospider

Rule Path
Disallow /

open text index robot

Rule Path
Disallow /

open web analytics bot

Rule Path
Disallow /

openbot

Rule Path
Disallow /

opencalaissemanticproxy

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openfind data gatherer

Rule Path
Disallow /

openfind daten sammler

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

openwebspider

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

orb search

Rule Path
Disallow /

orgbybot

Rule Path
Disallow /

osobot

Rule Path
Disallow /

ourbrowser

Rule Path
Disallow /

owncloud server crawler

Rule Path
Disallow /

pack rat

Rule Path
Disallow /

page2rss

Rule Path
Disallow /

pageboy

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

page_verifier

Rule Path
Disallow /

panopta.com

Rule Path
Disallow /

panscient web crawler

Rule Path
Disallow /

papa foto

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

parasite

Rule Path
Disallow /

parchbot

Rule Path
Disallow /

parsijoo

Rule Path
Disallow /

patric

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

paypal ipn

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pear.php.net

Rule Path
Disallow /

peeplo screenshot bot

Rule Path
Disallow /

peepowbot

Rule Path
Disallow /

peerindex

Rule Path
Disallow /

peew

Rule Path
Disallow /

pegasus

Rule Path
Disallow /

percolatecrawler

Rule Path
Disallow /

perlcrawler 1.0

Rule Path
Disallow /

perman

Rule Path
Disallow /

pgp key agent

Rule Path
Disallow /

phantom

Rule Path
Disallow /

phpdig

Rule Path
Disallow /

php\/

Rule Path
Disallow /

piltdownman

Rule Path
Disallow /

pimptrain.com’s robot

Rule Path
Disallow /

pingdom.com_bot

Rule Path
Disallow /

pinterest

Rule Path
Disallow /

pioneer

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

playmusic\.com

Rule Path
Disallow /

playstarmusic\.com

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

plumtreewebaccessor

Rule Path
Disallow /

pmoz.info odp link checker

Rule Path
Disallow /

pompos

Rule Path
Disallow /

poppi

Rule Path
Disallow /

popular iconoclast

Rule Path
Disallow /

portal juice spider

Rule Path
Disallow /

portalb spider

Rule Path
Disallow /

postpost

Rule Path
Disallow /

powermarks

Rule Path
Disallow /

procogbot

Rule Path
Disallow /

procogseobot

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

proximic

Rule Path
Disallow /

psbot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

qihoobot

Rule Path
Disallow /

qirina hurdler

Rule Path
Disallow /

qseero

Rule Path
Disallow /

qualidator.com bot

Rule Path
Disallow /

qualidator.com siteanalyzer 1.0

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

queryn metasuche

Rule Path
Disallow /

queryn.metasearch

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

quickobot

Rule Path
Disallow /

r6 bot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

radar-bot

Rule Path
Disallow /

radiation retriever 1.1

Rule Path
Disallow /

radiation retriever 1/1

Rule Path
Disallow /

rambler

Rule Path
Disallow /

rankurbot

Rule Path
Disallow /

raven search

Rule Path
Disallow /

rbse spider

Rule Path
Disallow /

readability

Rule Path
Disallow /

readpaper

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

realdownload/4.0.0.40

Rule Path
Disallow /

realdownload/4.0.0.41

Rule Path
Disallow /

realdownload/4.0.0.42

Rule Path
Disallow /

redalert|robozilla

Rule Path
Disallow /

reget

Rule Path
Disallow /

repec.link.checker

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

repomonkey kã¶der & tackle/v1.01

Rule Path
Disallow /

resume robot

Rule Path
Disallow /

rixbot

Rule Path
Disallow /

rma

Rule Path
Disallow /

road runner: the imagescape robot

Rule Path
Disallow /

roadhouse crawling system

Rule Path
Disallow /

robbie the robot

Rule Path
Disallow /

robocrawl spider

Rule Path
Disallow /

robofox

Rule Path
Disallow /

robot

Rule Path
Disallow /

robot francoroute

Rule Path
Disallow /

robots

Rule Path
Disallow /

robots_tester

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ronzoobot

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

rpt\-httpclient\/0.3-3e

Rule Path
Disallow /

rss

Rule Path
Disallow /

rssmicro.com rss/atom feed robot

Rule Path
Disallow /

ruky-roboter

Rule Path
Disallow /

rules

Rule Path
Disallow /

ryzecrawler

Rule Path
Disallow /

safetynet robot

Rule Path
Disallow /

sai crawler

Rule Path
Disallow /

sanszbot

Rule Path
Disallow /

sbider

Rule Path
Disallow /

sbsearch

Rule Path
Disallow /

scan4mail

Rule Path
Disallow /

scarlett

Rule Path
Disallow /

scientificcommons

Rule Path
Disallow /

scirus

Rule Path
Disallow /

scooter

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

screenerbot crawler

Rule Path
Disallow /

search.aus-au.com

Rule Path
Disallow /

search.kumkie.com

Rule Path
Disallow /

search17bot

Rule Path
Disallow /

searchmetrics

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

searchprocess

Rule Path
Disallow /

securityresearchbot

Rule Path
Disallow /

seegnifybot

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

semager

Rule Path
Disallow /

semantifire

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

senrigan

Rule Path
Disallow /

seobility

Rule Path
Disallow /

seocheckbot

Rule Path
Disallow /

seodat

Rule Path
Disallow /

seoengbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

setoozbot

Rule Path
Disallow /

sg-scout

Rule Path
Disallow /

shagseeker

Rule Path
Disallow /

shai’hulud

Rule Path
Disallow /

shareaholicbot

Rule Path
Disallow /

shelob

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

shoutcast

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

sift

Rule Path
Disallow /

simmany robot ver1.0

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

site searcher

Rule Path
Disallow /

site valet

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sitecondor

Rule Path
Disallow /

sitedomain-bot

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitetech-rover

Rule Path
Disallow /

skymob.com

Rule Path
Disallow /

slcrawler

Rule Path
Disallow /

sleek

Rule Path
Disallow /

slurp

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smart spider

Rule Path
Disallow /

smart.apnoti.com robot

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

smartdownload/1.2.76 (win32; 1. april 1999)

Rule Path
Disallow /

smartdownload/1.2.76 (win32; apr 1 1999)

Rule Path
Disallow /

smartdownload/1.2.77 (win32, 17 august 1999)

Rule Path
Disallow /

smartdownload/1.2.77 (win32, 19 juni 2001)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; 1. februar 2000)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; aug 17 1999)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; feb 1 2000)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; jun 19 2001)

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

sniffrss

Rule Path
Disallow /

snooper

Rule Path
Disallow /

snooper/b97_01

Rule Path
Disallow /

snoopy

Rule Path
Disallow /

socialbm_bot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

solbot

Rule Path
Disallow /

solbot/1.0 lwp/5.07

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

sootle

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sougou+web+spider

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spanner/1.0 (linux 2.0.27 i586)

Rule Path
Disallow /

spbot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

spider

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

spiderline crawler

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

spiderview

Rule Path
Disallow /

spiderview(tm)

Rule Path
Disallow /

spider_monkey

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

spry wizard robot

Rule Path
Disallow /

spyder3.microsys.com

Rule Path
Disallow /

sqworm/2.9.85-beta (beta_release; 20011115-775, i686-pc-linux

Rule Path
Disallow /

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux

Rule Path
Disallow /

ssl-crawler

Rule Path
Disallow /

sslbot

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

statoolsbot

Rule Path
Disallow /

steeler

Rule Path
Disallow /

stichwort density/0.9

Rule Path
Disallow /

strider

Rule Path
Disallow /

strokebot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

suke

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

suntek search engine

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superbot/3.0 (win32)

Rule Path
Disallow /

superbot/3.1 (win32)

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

superhttp/1.0

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surphace scout

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

sven

Rule Path
Disallow /

swebot

Rule Path
Disallow /

sygol

Rule Path
Disallow /

sygolbot

Rule Path
Disallow /

symfony spider

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

tach black widow

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

tailrank

Rule Path
Disallow /

takeout

Rule Path
Disallow /

taptubot

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

tarspider

Rule Path
Disallow /

tcl w3 robot

Rule Path
Disallow /

techbot

Rule Path
Disallow /

technoratibot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

teleport pro/1.29

Rule Path
Disallow /

teleport pro/1.29.1590

Rule Path
Disallow /

teleport pro/1.29.1634

Rule Path
Disallow /

teleport pro/1.29.1718

Rule Path
Disallow /

teleport pro/1.29.1820

Rule Path
Disallow /

teleport pro/1.29.1847

Rule Path
Disallow /

teleport(\s|\+)pro

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

templeton

Rule Path
Disallow /

teoma

Rule Path
Disallow /

teomatechnologies

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

the jubii indexing robot

Rule Path
Disallow /

the northstar robot

Rule Path
Disallow /

the nwi robot

Rule Path
Disallow /

the peregrinator

Rule Path
Disallow /

the python robot

Rule Path
Disallow /

the tkwww robot

Rule Path
Disallow /

the web moose

Rule Path
Disallow /

the web wombat

Rule Path
Disallow /

the webfoot robot

Rule Path
Disallow /

the world wide web wanderer

Rule Path
Disallow /

the world wide web worm

Rule Path
Disallow /

the.intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

thumbnail.cz robot

Rule Path
Disallow /

thumbshots-bot

Rule Path
Disallow /

thumbshots-de-bot

Rule Path
Disallow /

thumbshots.ru

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

tineye

Rule Path
Disallow /

titan

Rule Path
Disallow /

titin

Rule Path
Disallow /

tlspider

Rule Path
Disallow /

tocrawl / urldispatcher

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

tomtom places company search

Rule Path
Disallow /

topicbot

Rule Path
Disallow /

toread-crawler

Rule Path
Disallow /

touche

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

true_robot/1.0

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

turnitinbot/1.5

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twikle

Rule Path
Disallow /

twingly recon

Rule Path
Disallow /

t\-h\-u\-n\-d\-e\-r\-s\-t\-o\-n\-e

Rule Path
Disallow /

uaslinkchecker

Rule Path
Disallow /

ucsd

Rule Path
Disallow /

ucsd crawl

Rule Path
Disallow /

udmsearch

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

umbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

unwindfetchor

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

updownerbot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

uptimedog

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

url check

Rule Path
Disallow /

url control

Rule Path
Disallow /

url spider pro

Rule Path
Disallow /

url2file

Rule Path
Disallow /

urlaliasbuilder

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

urlck/1.2.3

Rule Path
Disallow /

urlfan-bot

Rule Path
Disallow /

urlfilebot (urlbot)

Rule Path
Disallow /

urllib

Rule Path
Disallow /

urlspiderpro

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

urly warnung

Rule Path
Disallow /

urly.warning

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

validator

Rule Path
Disallow /

valkyrie

Rule Path
Disallow /

valkyrie/1.0 libwww-perl/0.40

Rule Path
Disallow /

vbseo

Rule Path
Disallow /

vci

Rule Path
Disallow /

vci webviewer vci webviewer win32

Rule Path
Disallow /

vedma

Rule Path
Disallow /

verticrawl

Rule Path
Disallow /

victoria

Rule Path
Disallow /

videosurf_bot

Rule Path
Disallow /

virus[_+]detector

Rule Path
Disallow /

visbot

Rule Path
Disallow /

vision-search

Rule Path
Disallow /

void-bot

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voila

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

vwbot

Rule Path
Disallow /

w3c\-checklink

Rule Path
Disallow /

w3m2

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

w@pspider by wap4.com

Rule Path
Disallow /

walhello appie

Rule Path
Disallow /

wallpaper (alias crawlpaper)

Rule Path
Disallow /

wanadoo

Rule Path
Disallow /

wasalive-bot

Rule Path
Disallow /

watchmouse

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

web core / roots

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

web sucker

Rule Path
Disallow /

web(\s|\+)downloader

Rule Path
Disallow /

web-monitoring

Rule Path
Disallow /

web-sniffer

Rule Path
Disallow /

web-sucker

Rule Path
Disallow /

web.image.collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webauto/3.40 (win98; i)

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webbandit web spider

Rule Path
Disallow /

webbandit/3.50

Rule Path
Disallow /

webcapture 2,0

Rule Path
Disallow /

webcapture 2.0

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webcloner

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcookies

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcopier v.2.2

Rule Path
Disallow /

webcopier v2.5

Rule Path
Disallow /

webcopier v2.6

Rule Path
Disallow /

webcopier v2.7a

Rule Path
Disallow /

webcopier v2.8

Rule Path
Disallow /

webcopier v3.0

Rule Path
Disallow /

webcopier v3.0.1

Rule Path
Disallow /

webcopier v3.2

Rule Path
Disallow /

webcopier v3.2a

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webcopy/

Rule Path
Disallow /

webcorp

Rule Path
Disallow /

webdatacentrebot

Rule Path
Disallow /

webemailextrac .*

Rule Path
Disallow /

webemailextrac.*

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webferret

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webfetch/2.1.0

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

webfetcher/0.8,

Rule Path
Disallow /

webgo is

Rule Path
Disallow /

webimages

Rule Path
Disallow /

webinator

Rule Path
Disallow /

webinatorbot

Rule Path
Disallow /

weblayers

Rule Path
Disallow /

weblayers/0.0

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

weblinker

Rule Path
Disallow /

weblinker/0.0 libwww-perl/0.1

Rule Path
Disallow /

weblog monitor

Rule Path
Disallow /

webmastercoffee

Rule Path
Disallow /

webmasterworld extractor

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webmetrics

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webmoose/0.0.0000

Rule Path
Disallow /

webnl

Rule Path
Disallow /

webquest

Rule Path
Disallow /

webrankspider

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webreaper v9.1 – www.otway.com/webreaper

Rule Path
Disallow /

webreaper v9.1 - www.otway.com / webreaper

Rule Path
Disallow /

webreaper v9.7 – www.webreaper.net

Rule Path
Disallow /

webreaper v9.7 - www.webreaper.net

Rule Path
Disallow /

webreaper v9.8 – www.webreaper.net

Rule Path
Disallow /

webreaper v9.8 - www.webreaper.net

Rule Path
Disallow /

webreaper vwebreaper v7.3 – www,otway.com/webreaper

Rule Path
Disallow /

webreaper vwebreaper v7.3 - www, otway.com / webreaper

Rule Path
Disallow /

webreaper [info@webreaper.net]

Rule Path
Disallow /

webreaper [webreaper@otway.com]

Rule Path
Disallow /

webs

Rule Path
Disallow /

webs@recruit.co.jp

Rule Path
Disallow /

websauger

Rule Path
Disallow /

websauger 1.20b

Rule Path
Disallow /

websauger 1.20j

Rule Path
Disallow /

websauger 1.20k

Rule Path
Disallow /

website extractor

Rule Path
Disallow /

website extractor (http://www.asona.org)

Rule Path
Disallow /

website quester

Rule Path
Disallow /

website quester – www.asona.org

Rule Path
Disallow /

website quester – www.esalesbiz.com/extra/

Rule Path
Disallow /

website quester - www.asona.org

Rule Path
Disallow /

website quester - www.esalesbiz.com/extra/

Rule Path
Disallow /

website-extraktor

Rule Path
Disallow /

website.quester

Rule Path
Disallow /

websnarf

Rule Path
Disallow /

webspider

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webster.pro

Rule Path
Disallow /

webstolperer

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webstripper/2.03

Rule Path
Disallow /

webstripper/2.10

Rule Path
Disallow /

webstripper/2.12

Rule Path
Disallow /

webstripper/2.13

Rule Path
Disallow /

webstripper/2.15

Rule Path
Disallow /

webstripper/2.16

Rule Path
Disallow /

webstripper/2.19

Rule Path
Disallow /

webthumbnail

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webvac/1.0

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webwalker

Rule Path
Disallow /

webwalker/1.10

Rule Path
Disallow /

webwatch

Rule Path
Disallow /

webwatch/robot_txtchecker

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzinger

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip/2.75 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/3.65 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/3.80 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

webzip/4.0 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/4.1 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/4.21

Rule Path
Disallow /

webzip/4.21 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/5.0

Rule Path
Disallow /

webzip/5.0 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/5.0 pr1 (http://www.spidersoft.com)

Rule Path
Disallow /

wesee

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget/1.4.0

Rule Path
Disallow /

wget/1.5.2

Rule Path
Disallow /

wget/1.5.3

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

wget/1.7

Rule Path
Disallow /

wget/1.8

Rule Path
Disallow /

wget/1.8.1

Rule Path
Disallow /

wget/1.8.1+cvs

Rule Path
Disallow /

wget/1.8.2

Rule Path
Disallow /

wget/1.9-beta

Rule Path
Disallow /

wget/1.5.2

Rule Path
Disallow /

wget/1.5.3

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

wget/1.7

Rule Path
Disallow /

wget/1.8

Rule Path
Disallow /

wget/1.8.1

Rule Path
Disallow /

wget/1.8.1 + cvs

Rule Path
Disallow /

wget/1.8.2

Rule Path
Disallow /

wget/1.9-beta

Rule Path
Disallow /

whatuseek winona

Rule Path
Disallow /

whoismindbot

Rule Path
Disallow /

whowhere robot

Rule Path
Disallow /

widow

Rule Path
Disallow /

wikiofeedbot

Rule Path
Disallow /

wikiwix-bot

Rule Path
Disallow /

wild ferret web hopper

Product Comment
wild ferret web hopper 1, #2, #3
Rule Path
Disallow /

willow internet crawler

Rule Path
Disallow /

willybot

Rule Path
Disallow /

winwebbot

Rule Path
Disallow /

wired digital

Rule Path
Disallow /

wired-digital-newsbot/1.5

Rule Path
Disallow /

witwe

Rule Path
Disallow /

wmcai_robot

Rule Path
Disallow /

woko

Rule Path
Disallow /

wordpress

Rule Path
Disallow /

woriobot

Rule Path
Disallow /

worm

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wsanalyzer

Rule Path
Disallow /

wscheck.com

Rule Path
Disallow /

www collector

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

www.freeloader.com.

Rule Path
Disallow /

www.gnip.com

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

wwwc ver 0.2.5

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

wwwwanderer v3.0

Rule Path
Disallow /

www\-mechanize

Rule Path
Disallow /

x28-job-bot

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

xaldon webspider 2.5.b3

Rule Path
Disallow /

xaldon_webspider

Rule Path
Disallow /

xenu’s

Rule Path
Disallow /

xenu’s link sleuth 1.1c

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xenu(\s|\+)link(\s|\+)sleuth

Rule Path
Disallow /

xget

Rule Path
Disallow /

xget/0.7

Rule Path
Disallow /

xmarksfetch

Rule Path
Disallow /

xml sitemaps generator

Rule Path
Disallow /

xrl

Rule Path
Disallow /

xyleme robot

Rule Path
Disallow /

y!j

Rule Path
Disallow /

yaanb

Rule Path
Disallow /

yacy

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yahoo! slurp china

Rule Path
Disallow /

yamanalab-robot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yanga

Rule Path
Disallow /

yasaklibot

Rule Path
Disallow /

yes

Rule Path
Disallow /

yesupbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yioopbot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

yowedobot

Rule Path
Disallow /

yrspider

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zeerchbot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zeus 11389 webster pro v2.9 win32

Rule Path
Disallow /

zeus 11652 webster pro v2.9 win32

Rule Path
Disallow /

zeus 18018 webster pro v2.9 win32

Rule Path
Disallow /

zeus 26378 webster pro v2.9 win32

Rule Path
Disallow /

zeus 30747 webster pro v2.9 win32

Rule Path
Disallow /

zeus 32297 webster pro v2.9 win32

Rule Path
Disallow /

zeus 39206 webster pro v2.9 win32

Rule Path
Disallow /

zeus 41641 webster pro v2.9 win32

Rule Path
Disallow /

zeus 44238 webster pro v2.9 win32

Rule Path
Disallow /

zeus 51070 webster pro v2.9 win32

Rule Path
Disallow /

zeus 51674 webster pro v2.9 win32

Rule Path
Disallow /

zeus 51837 webster pro v2.9 win32

Rule Path
Disallow /

zeus 63567 webster pro v2.9 win32

Rule Path
Disallow /

zeus 6694 webster pro v2.9 win32

Rule Path
Disallow /

zeus 71129 webster pro v2.9 win32

Rule Path
Disallow /

zeus 82016 webster pro v2.9 win32

Rule Path
Disallow /

zeus 82900 webster pro v2.9 win32

Rule Path
Disallow /

zeus 84842 webster pro v2.9 win32

Rule Path
Disallow /

zeus 90872 webster pro v2.9 win32

Rule Path
Disallow /

zeus 94934 webster pro v2.9 win32

Rule Path
Disallow /

zeus 95245 webster pro v2.9 win32

Rule Path
Disallow /

zeus 95351 webster pro v2.9 win32

Rule Path
Disallow /

zeus 97371 webster pro v2.9 win32

Rule Path
Disallow /

zeus link scout

Rule Path
Disallow /

zibber-v0.1(www.zibb.com/crawler/)

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

zookabot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://kotly.com/1_index_sitemap.xml
Back to top

Comments

  • Bad Robots block
  • User-agent: AdsBot-Google
  • User-agent: baiduspider
  • User-agent: Baiduspider-image
  • User-agent: Baiduspider-news
  • User-agent: Baiduspider-video
  • User-agent: Bingbot
  • User-agent: BingPreview
  • User-agent: Feedfetcher-Google
  • User-agent: feedfetcher\-google
  • User-agent: google
  • User-agent: Googlebot
  • User-agent: Googlebot-Image
  • User-agent: Googlebot-News
  • User-agent: Mediapartners-Google
  • User-agent: mediapartners\-google
  • User-agent: seznambot
  • User-agent: WebCrawler/3.0 Robot libwww/5.0a
  • User-agent: yahoo
  • User-agent: Yahoo Pipes 1.0
  • User-agent: Yahoo!
  • User-agent: Yahoo! Slurp
  • User-agent: Yandex
  • User-agent: YandexAntivirus
  • User-agent: YandexBlogs
  • User-agent: YandexBot
  • User-agent: YandexCatalog
  • User-agent: YandexDirect
  • User-agent: YandexFavicons
  • User-agent: YandexImages
  • User-agent: YandexMarket
  • User-agent: YandexMedia
  • User-agent: YandexNews
  • User-agent: YandexPagechecker
  • User-agent: YandexVideo
  • User-agent: YandexWebmaster
  • User-agent: YandexZakladki
Back to top

Warnings

  • 28 invalid lines.
Back to top