Log In Sign Up

neteron.ru
robots.txt

Robots Exclusion Standard data for neteron.ru

Resource Scan

Scan Details

Site Domain neteron.ru
Base Domain neteron.ru
Scan Status Ok
Last Scan2024-11-03T14:30:20+00:00
Next Scan 2024-12-03T14:30:20+00:00

Last Scan

Scanned2024-11-03T14:30:20+00:00
URL https://neteron.ru/robots.txt
Domain IPs 141.8.192.102, 141.8.196.46, 2a0a:2b43:7:9fdd::
Response IP 141.8.192.102
Found Yes
Hash f138a82158b4b4d6f1119157f677f7dfe769af9aa23bc06641538ef49f7aa865
SimHash 51818753abf7

Groups

*

Rule Path
Allow /sitemap.php
Disallow /index.php?
Disallow /*index.php*
Disallow /admin
Disallow /api
Disallow /applications*
Disallow /datastore
Disallow /dev
Disallow /oauth
Disallow /plugins
Disallow /system
Disallow /uploads
Disallow /vendor
Disallow /404error.php
Disallow /Credits.txt
Disallow /error.php
Disallow /login/
Disallow /logout
Disallow /register
Disallow /lostpassword
Disallow /privacy
Disallow /cookies
Disallow /guidelines
Disallow /terms
Disallow /online
Disallow /staff
Disallow /contact
Disallow /discover/
Disallow /profile/
Disallow /*announcement
Disallow /search
Disallow /*profile*
Disallow /profile
Disallow /*login
Disallow /rss*
Disallow /activity
Disallow /new-content
Disallow /*promote
Disallow /ourpicks
Disallow /leaderboard
Disallow /pastleaders
Disallow /topmembers
Disallow /terms*
Disallow /*do%3D*
Disallow /*sort%3D*
Disallow /*sortby%3D*
Disallow /*csrf%3D*
Disallow /*csrfKey%3D*
Disallow */?tab=*
Disallow */?_fromLogin=*
Disallow */?_fromLogout=*
Disallow */submit
Disallow */create
Disallow */edit
Disallow /terms
Disallow /*terms*
Disallow /?app*
Disallow */?app
Disallow /discover/
Disallow /discover/*
Disallow /forum/applications/*
Disallow /forum/discover/*
Allow /uploads/monthly_*_*/*

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

Rule Path
Disallow /profile

yandexbot

Rule Path
Disallow /terms

yandexbot

Rule Path
Disallow /?app

yandexbot

Rule Path
Disallow /index.php

yandexbot

Rule Path
Disallow /search

yandexbot

Rule Path
Disallow /login

abcdatos botlink

Rule Path
Disallow /

abcdee

Rule Path
Disallow /

accessdiver

Rule Path
Disallow /

acme.spider

Rule Path
Disallow /

ahoy! the homepage finder

Rule Path
Disallow /

alexa

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alkaline

Rule Path
Disallow /

ananzi

Rule Path
Disallow /

anthill

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

arale

Rule Path
Disallow /

araneo

Rule Path
Disallow /

araybot

Rule Path
Disallow /

archive

Rule Path
Disallow /

aretha

Rule Path
Disallow /

ariadne

Rule Path
Disallow /

arks

Rule Path
Disallow /

askjeeves

Rule Path
Disallow /

aspider (associative spider)

Rule Path
Disallow /

asterias

Rule Path
Disallow /

atn worldwide

Rule Path
Disallow /

atomz

Rule Path
Disallow /

atomz.com search robot

Rule Path
Disallow /

atv

Rule Path
Disallow /

auresys

Rule Path
Disallow /

a-online search

Rule Path
Disallow /

abachobot

Rule Path
Disallow /

abachobot (mozilla compatible)

Rule Path
Disallow /

abcdatos botlink/5.xx.xxx

Rule Path
Disallow /

aberja checkomat

Rule Path
Disallow /

abot/0.1 (abot; http://www.abot.com; abot@abot.com)

Rule Path
Disallow /

about/0.1libwww-perl/5.47

Rule Path
Disallow /

ace explorer

Rule Path
Disallow /

acoirobot

Rule Path
Disallow /

acoon robot v1.50.001

Rule Path
Disallow /

acoon robot v1.52 (http://www.acoon.de)

Rule Path
Disallow /

accoona

Rule Path
Disallow /

activebookmark 1.0

Rule Path
Disallow /

activeworlds/3.xx (xxx)

Rule Path
Disallow /

ad muncher v4.xx.x

Rule Path
Disallow /

aesop_com_spiderman

Rule Path
Disallow /

agadine/1.x.x (+http://www.agada.de)

Rule Path
Disallow /

agent-sharewareplazafilecheckbot/2.0+(+http://www.sharewareplaza.com)

Rule Path
Disallow /

agentname/0.1 libwww-perl/5.48

Rule Path
Disallow /

aladin/3.324

Rule Path
Disallow /

alkalinebot/1.3

Rule Path
Disallow /

alkalinebot/1.4 (1.4.0326.0 rtm)

Rule Path
Disallow /

aleksika spider/1.0 (+http://www.aleksika.com/)

Rule Path
Disallow /

allesklar/0.1 libwww-perl/5.46

Rule Path
Disallow /

alligator 1.31 (www.nearsoftware.com)

Rule Path
Disallow /

altavista intranet v2.0 avs eval search@freeit.com

Rule Path
Disallow /

altavista intranet v2.0 compaq altavista eval sveand@altavista.net

Rule Path
Disallow /

altavista intranet v2.0 evreka.com crawler@evreka.com

Rule Path
Disallow /

altavista v2.0b crawler@evreka.com

Rule Path
Disallow /

amfibibot

Rule Path
Disallow /

amfibibot/0.06 (amfibi web search; http://www.amfibi.com; agent@amfibi.com)

Rule Path
Disallow /

amibot

Rule Path
Disallow /

amiga-aweb/3.4.167se

Rule Path
Disallow /

amigavoyager/3.4.4 (morphos/ppc native)

Rule Path
Disallow /

annomille spider 0.1 alpha - http://www.annomille.it

Rule Path
Disallow /

anonymized by proxyos: http://www.megaproxy.com

Rule Path
Disallow /

anonymizer/1.1

Rule Path
Disallow /

answerchase x.0

Rule Path
Disallow /

answerchase prove x.0

Rule Path
Disallow /

antfresco/x.xx

Rule Path
Disallow /

anzwerscrawl/2.0 (anzwerscrawl@anzwers.com.au;engine)

Rule Path
Disallow /

antibot-v1.1.5/i586-linux-2.2

Rule Path
Disallow /

aplix http/1.0.1

Rule Path
Disallow /

aplix_sanyo_browser/1.x (japanese)

Rule Path
Disallow /

aplix_segasaturn_browser/1.x (japanese)

Rule Path
Disallow /

appie 1.1 (www.walhello.com)

Rule Path
Disallow /

aport

Rule Path
Disallow /

arachnoidea (arachnoidea@euroseek.com)

Rule Path
Disallow /

architextspider

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

argus/1.1 (nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)

Rule Path
Disallow /

asahina-antenna/1.x

Rule Path
Disallow /

asahina-antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)

Rule Path
Disallow /

ask.24x.info

Rule Path
Disallow /

aspseek/1.2.x

Rule Path
Disallow /

aspseek/1.2.xx

Rule Path
Disallow /

aspseek/1.2.xxpre

Rule Path
Disallow /

aspseek/1.2.xa

Rule Path
Disallow /

aspseek/1.2.5

Rule Path
Disallow /

aspseek/1.2.9d

Rule Path
Disallow /

assort/0.10

Rule Path
Disallow /

asterias/2.0

Rule Path
Disallow /

atlocalbot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)

Rule Path
Disallow /

atomz/1.0

Rule Path
Disallow /

atspider/1.0

Rule Path
Disallow /

augurfind

Rule Path
Disallow /

augurnfind v-1.x

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

autohttp

Rule Path
Disallow /

av fetch 1.0

Rule Path
Disallow /

avsearch-1.0(peter.turney@nrc.ca)

Rule Path
Disallow /

avsearch-2.0-fusionidx-14-competitorwebsites

Rule Path
Disallow /

avsearch-3.0(altavista/avc)

Rule Path
Disallow /

axmorobot - crawling your site for better indexing on www.axmo.com search engine.

Rule Path
Disallow /

b2w/0.1

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backrub

Rule Path
Disallow /

baldgorilla

Rule Path
Disallow /

bbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

big brother

Rule Path
Disallow /

bjaaland

Rule Path
Disallow /

black hole

Rule Path
Disallow /

bloodhound

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

boardviewer

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

borg-bot

Rule Path
Disallow /

botalot

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

boxseabot

Rule Path
Disallow /

bright.net caching robot

Rule Path
Disallow /

browse

Rule Path
Disallow /

bspider

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

burner

Rule Path
Disallow /

baboombot/1.x.x (+http://www.baboom.us)

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+ (+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

banbots/1.2 (spider@banbots.com)

Rule Path
Disallow /

barrahomecrawler (albertof@barrahome.org)

Rule Path
Disallow /

bdcindexer_2.6.2 (research@bdc)

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

bdncentral crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (x11; i; linux 2.0.44 i686)

Rule Path
Disallow /

beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)

Rule Path
Disallow /

beebwaredirectory/v0.01 - http://directory.beebware.co.uk linkchecker

Rule Path
Disallow /

big brother (http://pauillac.inria.fr/~fpottier/)

Rule Path
Disallow /

bigbrother/1.6e

Rule Path
Disallow /

bigcliquebot/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)

Rule Path
Disallow /

bilbo/2.3b-unix

Rule Path
Disallow /

billbot wjj@cs.cmu.edu

Rule Path
Disallow /

biyubi/x.x (sistema fenix; g11; familia toledo; es-mx)

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blaiz-bee/1.0 (+http://www.blaiz.net)

Rule Path
Disallow /

blitzbot@tricus.net

Rule Path
Disallow /

blitzbot@tricus.net (mozilla compatible)

Rule Path
Disallow /

blogbot/1.x

Rule Path
Disallow /

bloglines title fetch/1.0 (http://www.bloglines.com)

Rule Path
Disallow /

blogwatcher_spider/0.1 (http://www.lr.pi.titech.ac.jp/blogwatcher/)

Rule Path
Disallow /

bluefish 0.6 html editor

Rule Path
Disallow /

bmclient

Rule Path
Disallow /

bobby/4.0.x rpt-httpclient/0.3-3e

Rule Path
Disallow /

boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)

Rule Path
Disallow /

boitho.com-robot/1.x

Rule Path
Disallow /

boitho.com-robot/1.x (http://www.boitho.com/bot.html)

Rule Path
Disallow /

bookmark buddy bookmark checker (http://www.bookmarkbuddy.net/)

Rule Path
Disallow /

bookmark renewal check agent [http://www.bookmark.ne.jp/]

Rule Path
Disallow /

bot mailto:craftbot@yahoo.com

Rule Path
Disallow /

bravobrian bstop.bravobrian.it

Rule Path
Disallow /

bravobrian spiderengine marcopolo

Rule Path
Disallow /

bsdseek/1.0

Rule Path
Disallow /

bstop.bravobrian.it agent detector

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bumblebee@relevare.com

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

buscaplus robi/1.0 (http://www.buscaplus.com/robi/)

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

calif

Rule Path
Disallow /

cassandra

Rule Path
Disallow /

catalog

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cerb

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypicker /1.0

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

christcrawler.com

Rule Path
Disallow /

churl

Rule Path
Disallow /

cienciaficcion.net

Rule Path
Disallow /

clickgarden

Rule Path
Disallow /

cmc

Rule Path
Disallow /

cmc/0.01

Rule Path
Disallow /

collective

Rule Path
Disallow /

collector

Rule Path
Disallow /

combine system

Rule Path
Disallow /

computingsite robi/1.0

Rule Path
Disallow /

confuzzledbot

Rule Path
Disallow /

coolbot

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

crawl

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

ctphotobook

Rule Path
Disallow /

cusco

Rule Path
Disallow /

cyberspyder link test

Rule Path
Disallow /

cydralspider

Rule Path
Disallow /

cancer information and support international;

Rule Path
Disallow /

carnegie_mellon_university_webcrawler,http://www.andrew.cmu.edu/~brgordon/webbot/index.html

Rule Path
Disallow /

carnegie_mellon_university_research_webbot-->please read-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html

Rule Path
Disallow /

ceramic tile installation guide (http://www.floorstransformed.com)

Rule Path
Disallow /

cg-eye interactive

Rule Path
Disallow /

charon/1.x (amiga)

Rule Path
Disallow /

checkbot/1.xx lwp/5.xx

Rule Path
Disallow /

checklinks/1.x.x

Rule Path
Disallow /

checkurl

Rule Path
Disallow /

checkweb

Rule Path
Disallow /

christcrawler 2.0

Rule Path
Disallow /

cipinetbot (http://www.cipinet.com/bot.html)

Rule Path
Disallow /

cj spider/

Rule Path
Disallow /

clariabot/1.0

Rule Path
Disallow /

claymont.com

Rule Path
Disallow /

clushbot/2.1 (+http://www.clush.com/bot.html)

Rule Path
Disallow /

clushbot/3.3-binaryfury (+http://www.clush.com/bot.html)

Rule Path
Disallow /

coldfusion

Rule Path
Disallow /

coldfusion (bookmarktracker.com)

Rule Path
Disallow /

collage.cgi/1.82

Rule Path
Disallow /

combine/0.0

Rule Path
Disallow /

combine/x.0

Rule Path
Disallow /

commerce browser center

Rule Path
Disallow /

contact

Rule Path
Disallow /

contype

Rule Path
Disallow /

converacrawler/0.2

Rule Path
Disallow /

convera internet spider v6.x

Rule Path
Disallow /

converamultimediacrawler/0.1 (+http://www.authoritativeweb.com/crawl)

Rule Path
Disallow /

cosmos/0.8_(robot@xyleme.com)

Rule Path
Disallow /

cosmos/0.9_(robot@xyleme.com)

Rule Path
Disallow /

cowbot-0.1 (nhn corp. / +82-2-3011-1954 / nhnbot@naver.com)

Rule Path
Disallow /

cowbot-0.1.x (nhn corp. / +82-2-3011-1954 / nhnbot@naver.com)

Rule Path
Disallow /

crawlconvera0.1 (crawlconvera@yahoo.com)

Rule Path
Disallow /

crawler

Rule Path
Disallow /

crawler (cometsearch@cometsystems.com)

Rule Path
Disallow /

crawler admin@crawler.de

Rule Path
Disallow /

crawler v 0.2.x admin@crawler.de

Rule Path
Disallow /

crawler@alexa.com

Rule Path
Disallow /

crawlerboy pinpoint.com

Rule Path
Disallow /

creativecommons/0.06-dev (nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

Rule Path
Disallow /

croccrawler vx.3 [en] (http://www.croccrawler.com) (x11; i; linux 2.0.44 i686)

Rule Path
Disallow /

cuam ver0.050bx

Rule Path
Disallow /

cuasarbot/0.9b http://www.cuasar.com/spider_beta/

Rule Path
Disallow /

curl/7.7.1 (i386--freebsd4.3) libcurl 7.7.1 (ssl 0.9.6) (ipv6 enabled)

Rule Path
Disallow /

curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (openssl 0.9.6)

Rule Path
Disallow /

curl/7.9.x (win32) libcurl 7.9.x

Rule Path
Disallow /

curryguide sitescan 1.1

Rule Path
Disallow /

custo x.x (www.netwu.com)

Rule Path
Disallow /

custom spider www.bisnisseek.com /1.0

Rule Path
Disallow /

cyberdog/2.0 (macintosh; 68k)

Rule Path
Disallow /

cyberspyder link test/2.1.12 (admin@mspennyworth.com)

Rule Path
Disallow /

cydralspider/1.x (cydral web image search; http://www.cydral.com)

Rule Path
Disallow /

da

Rule Path
Disallow /

desert realm spider

Rule Path
Disallow /

deweb(c) katalog/index

Rule Path
Disallow /

die blinde kuh

Rule Path
Disallow /

dienstspider

Rule Path
Disallow /

digger

Rule Path
Disallow /

digimarc marcspider

Rule Path
Disallow /

digimarc marcspider/cgi

Rule Path
Disallow /

digital integrity robot

Rule Path
Disallow /

diibot

Rule Path
Disallow /

direct hit grabber

Rule Path
Disallow /

discofinder

Rule Path
Disallow /

dnabot

Rule Path
Disallow /

download express

Rule Path
Disallow /

downloader

Rule Path
Disallow /

downloadit

Rule Path
Disallow /

dragonbot

Rule Path
Disallow /

dwcp (dridus' web cataloging project)

Rule Path
Disallow /

da 3.5 (www.lidan.com)

Rule Path
Disallow /

da 4.0

Rule Path
Disallow /

da 4.0 (www.downloadaccelerator.com)

Rule Path
Disallow /

da 5.0

Rule Path
Disallow /

da 7.0

Rule Path
Disallow /

dart communications powertcp

Rule Path
Disallow /

dataparksearch/4.xx (http://www.dataparksearch.org/)

Rule Path
Disallow /

daviesbot/1.7 (www.wholeweb.net)

Rule Path
Disallow /

dbdig(http://www.prairielandconsulting.com)

Rule Path
Disallow /

dbrowse 1.4b

Rule Path
Disallow /

dbrowse 1.4d

Rule Path
Disallow /

dc-sakura/x.xx

Rule Path
Disallow /

dcsbot/1.1

Rule Path
Disallow /

ddd

Rule Path
Disallow /

dds explorer v1.0 beta

Rule Path
Disallow /

deadlinkcheck/0.4.0 libwww-perl/5.xx

Rule Path
Disallow /

deepak-usc/isi

Rule Path
Disallow /

deepindex

Rule Path
Disallow /

deepindex ( http://www.zetbot.com )

Rule Path
Disallow /

deepindex (www.en.deepindex.com)

Rule Path
Disallow /

deepindexer.ca

Rule Path
Disallow /

demo bot dot 16b

Rule Path
Disallow /

demo bot z 16b

Rule Path
Disallow /

denmex websearch (http://search.denmex.com)

Rule Path
Disallow /

devcomponents.com htmldocument object

Rule Path
Disallow /

diagem/1.1 (http://www.skyrocket.gr.jp/diagem.html)

Rule Path
Disallow /

diamond/1.0

Rule Path
Disallow /

diamondbot

Rule Path
Disallow /

digger/1.0 jdk/1.3.0rc3

Rule Path
Disallow /

digout4u

Rule Path
Disallow /

dillo/0.x.x

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

diibot/1.2

Rule Path
Disallow /

dlman

Rule Path
Disallow /

dloader(naverrobot)/1.0

Rule Path
Disallow /

docomo/1.0/nxxxi/c10

Rule Path
Disallow /

docomo/1.0/nxxxi/c10/tb

Rule Path
Disallow /

docomo/1.0/p502i/c10 (google chtml proxy/1.0)

Rule Path
Disallow

docomo/2.0 p900iv(c100;tb;w24h11)

Rule Path
Disallow /

doczilla/1.0 (windows; u; winnt4.0; en-us; rv:1.0.0) gecko/20020804

Rule Path
Disallow /

download demon/3.x.x.x

Rule Path
Disallow /

download express 1.0

Rule Path
Disallow /

download master

Rule Path
Disallow /

download wonder

Rule Path
Disallow /

download-tipp linkcheck (http://download-tipp.de/)

Rule Path
Disallow /

download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/)

Rule Path
Disallow /

drecombot/1.0 (http://career.drecom.jp/bot.html)

Rule Path
Disallow /

dsurf15a 01

Rule Path
Disallow /

dsurf15a 71

Rule Path
Disallow /

dsurf15a 81

Rule Path
Disallow /

dsurf15a va

Rule Path
Disallow /

dtsearchspider

Rule Path
Disallow /

dual proxy

Rule Path
Disallow /

dumbot(version 0.1 beta)

Rule Path
Disallow /

dumbot(version 0.1 beta - dumbfind.com)

Rule Path
Disallow /

ebiness

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

e-collector

Rule Path
Disallow /

educate

Rule Path
Disallow /

efp@gmx\.net

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

eit link verifier robot

Rule Path
Disallow /

elfinbot

Rule Path
Disallow /

emacs-w3 search engine

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

esculapio

Rule Path
Disallow /

esther

Rule Path
Disallow /

evliya celebi

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html)

Rule Path
Disallow /

earthcom.info/1.x [www.earthcom.info]

Rule Path
Disallow /

earthcom.info/1.xbeta [www.earthcom.info]

Rule Path
Disallow /

easydl/3.xx

Rule Path
Disallow /

easydl/3.xx http://keywen.com/encyclopedia/bot

Rule Path
Disallow /

ebrowse 1.4b

Rule Path
Disallow /

ecatch/3.0

Rule Path
Disallow /

educate search vxb

Rule Path
Disallow /

echo!/2.0

Rule Path
Disallow /

egothor/3.0a (+http://www.xdefine.org/robot.html)

Rule Path
Disallow /

egotobot/4.8 (+http://www.egoto.com/about.htm)

Rule Path
Disallow /

eldos timelyweb/3.x

Rule Path
Disallow /

elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)

Rule Path
Disallow /

elinks (0.x.x; linux 2.4.20 i586; 132x60)

Rule Path
Disallow /

elinks/0.x.x (textmode; netbsd 1.6.2 sparc; 132x43)

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf 1.00

Rule Path
Disallow /

emeraldshield.com webbot

Rule Path
Disallow /

emeraldshield.com webbot (http://www.emeraldshield.com/webbot.aspx)

Rule Path
Disallow /

enfish tracker

Rule Path
Disallow /

enterprise_search/1.0

Rule Path
Disallow /

enterprise_search/1.0.xxx

Rule Path
Disallow /

enterprise_search/1.00.xxx;mssql (http://www.innerprise.net/es-spider.asp)

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

es.net_crawler/2.0 (http://search.innerprise.net/)

Rule Path
Disallow /

eseek-larbin_2.6.2 (crawler@exactseek.com)

Rule Path
Disallow /

esismartspider

Rule Path
Disallow /

estylesearch 4 (compatible; msie 6.0; windows nt 5.0)

Rule Path
Disallow /

esurf15a 15

Rule Path
Disallow /

exalead ng/mimelive client (convert/http/0.120)

Rule Path
Disallow /

exactseek crawler/0.1

Rule Path
Disallow /

exactseek-crawler-2.63 (crawler@exactseek.com)

Rule Path
Disallow /

excalibur internet spider v6.5.4

Rule Path
Disallow /

experimentalhenrythemiragorobot

Rule Path
Disallow /

expired domain sleuth

Rule Path
Disallow /

express webpictures (www.express-soft.com)

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyecatcher (download-tipp.de)/1.0

Rule Path
Disallow /

fastcrawler

Rule Path
Disallow /

felix ide

Rule Path
Disallow /

fetchrover

Rule Path
Disallow /

fido

Rule Path
Disallow /

fish search

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flickbot

Rule Path
Disallow /

fluid dynamics search engine robot

Rule Path
Disallow /

foobot

Rule Path
Disallow /

fouineur

Rule Path
Disallow /

franklin

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

full

Rule Path
Disallow /

funnelweb

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

fastbug http://www.ay-up.com

Rule Path
Disallow /

fast crawler gold edition

Rule Path
Disallow /

fast enterprise crawler 6 (experimental)

Rule Path
Disallow /

fast enterprise crawler 6 used by cobra development (admin@fastsearch.com)

Rule Path
Disallow /

fast enterprise crawler 6 used by fast (fast)

Rule Path
Disallow /

fast enterprise crawler/6 (www.fastsearch.com)

Rule Path
Disallow /

fast firstpage retriever (compatible; msie 5.5; mozilla/4.0)

Rule Path
Disallow /

fast partnersite crawler

Rule Path
Disallow /

fastcrawler 3.0.1 (crawler@1klik.dk)

Rule Path
Disallow /

fast-webcrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

Rule Path
Disallow /

fast-webcrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

Rule Path
Disallow /

fast-webcrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

Rule Path
Disallow /

fast-webcrawler/2.2.10 (multimedia search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

Rule Path
Disallow /

fast-webcrawler/3.2 test

Rule Path
Disallow /

fast-webcrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)

Rule Path
Disallow /

fast-webcrawler/3.4/nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)

Rule Path
Disallow /

fast-webcrawler/3.4/partnersite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)

Rule Path
Disallow /

fast-webcrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler)

Rule Path
Disallow /

fast-webcrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

Rule Path
Disallow /

fast-webcrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

Rule Path
Disallow /

fast-webcrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

Rule Path
Disallow /

fast-webcrawler/3.8/fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

Rule Path
Disallow /

fast-webcrawler/3.6/firstpage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)

Rule Path
Disallow /

fast-webcrawler/3.7/firstpage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)

Rule Path
Disallow /

fast-webcrawler/3.x multimedia

Rule Path
Disallow /

fast-webcrawler/3.x multimedia (mm dash crawler at fast dot no)

Rule Path
Disallow /

fastsearch web crawler for verizon superpages (kevin.watters@fastsearch.com)

Rule Path
Disallow /

favorites sweeper v.2.03

Rule Path
Disallow /

faxobot/1.0

Rule Path
Disallow /

felix - mixcat crawler (+http://mixcat.com)

Rule Path
Disallow /

ffc trap door spider

Rule Path
Disallow /

filangy/0.01-beta (filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)

Rule Path
Disallow /

filangy/1.0x (filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)

Rule Path
Disallow /

finebot

Rule Path
Disallow /

findanisp.com_isp_finder_v99a

Rule Path
Disallow /

findlinks/0.xxx (+http://wortschatz.uni-leipzig.de/findlinks/)

Rule Path
Disallow /

firefly/1.0

Rule Path
Disallow /

firefly/1.0 (compatible; mozilla 4.0; msie 5.5)

Rule Path
Disallow /

firstgov.gov search - poc:firstgov.webmasters@gsa.gov

Rule Path
Disallow /

firstsbot

Rule Path
Disallow /

flickbot 2.0 rpt-httpclient/0.3-3

Rule Path
Disallow /

flunky

Rule Path
Disallow /

fly/6.01 libwww/4.0d

Rule Path
Disallow /

focusedsampler/1.0

Rule Path
Disallow /

francis/1.0 (francis@neomo.de http://www.neomo.de/)

Rule Path
Disallow /

franklin locator 1.8

Rule Path
Disallow /

freefind.com-sitesearchengine/1.0 (http://freefind.com; spiderinfo@freefind.com)

Rule Path
Disallow /

fsurf15a 01

Rule Path
Disallow /

full web bot 0416b

Rule Path
Disallow /

full web bot 0516b

Rule Path
Disallow /

full web bot 2816b

Rule Path
Disallow /

fyberspider (+http://www.fybersearch.com/fyberspider.php)

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

gammaspider, focusedcrawler

Rule Path
Disallow /

gazz

Rule Path
Disallow /

gcreep

Rule Path
Disallow /

getright

Rule Path
Disallow /

getright/4.2

Rule Path
Disallow /

getterroboplus puu

Rule Path
Disallow /

geturl

Rule Path
Disallow /

getwebpage

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

golem

Rule Path
Disallow /

gozilla

Rule Path
Disallow /

grab

Rule Path
Disallow /

grafula

Rule Path
Disallow /

grapnel/0.01 experiment

Rule Path
Disallow /

green

Rule Path
Disallow /

griffon

Rule Path
Disallow /

gromit

Rule Path
Disallow /

grub

Rule Path
Disallow /

grub.org

Rule Path
Disallow /

gulper bot

Rule Path
Disallow /

gagglebot

Rule Path
Disallow /

gais robot/1.0b2

Rule Path
Disallow /

gaisbot/3.0+(indexer@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)

Rule Path
Disallow /

galaxybot/1.0 (http://www.galaxy.com/galaxybot.html)

Rule Path
Disallow /

gammaspider/1.0

Rule Path
Disallow /

gamespy_arcade, gamespyhttp/1.0

Rule Path
Disallow /

gazz/x.x (gazz@nttrd.com)

Rule Path
Disallow /

geckobot

Rule Path
Disallow /

generic_crawler/01.0217/

Rule Path
Disallow /

geonabot 1.x; http://www.geona.com/

Rule Path
Disallow /

getbot

Rule Path
Disallow /

getright/3.3.3

Rule Path
Disallow /

getright/4.x

Rule Path
Disallow /

getright/4.x[a-e]

Rule Path
Disallow /

getright/4.5xx

Rule Path
Disallow /

ghostroutehunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net)

Rule Path
Disallow /

gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/)

Rule Path
Disallow /

gigabot/x.0

Rule Path
Disallow /

gigabot/2.0 (gigablast.com)

Rule Path
Disallow /

gigabotsitesearch/2.0 (sitesearch.gigablast.com)

Rule Path
Disallow /

gnodspider (www.gnod.net)

Rule Path
Disallow /

go!zilla 3.x (www.gozilla.com)

Rule Path
Disallow /

go!zilla/4.x.x.xx

Rule Path
Disallow /

go-ahead-got-it/1.1

Rule Path
Disallow /

goblin/0.9 (http://www.goguides.org/)

Rule Path
Disallow /

goblin/0.9.x (http://www.goguides.org/goblin-info.html)

Rule Path
Disallow /

goforit.com

Rule Path
Disallow /

goofer/0.2

Rule Path
Disallow /

grigorbot 0.8 (http://www.grigor.biz/bot.html)

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

grub crawler(http://www.grub.org)

Rule Path
Disallow /

gsa-crawler (enterprise; gid-01422; jplastiras@google.com)

Rule Path
Disallow /

gsa-crawler (enterprise; gid-01742;gsatesting@rediffmail.com)

Rule Path
Disallow /

gulliver/1.3

Rule Path
Disallow /

gulper web bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/link/gulperbot)

Rule Path
Disallow /

hambot

Rule Path
Disallow /

harvest

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

havindex

Rule Path
Disallow /

hi (html index) search

Rule Path
Disallow /

hku www octopus

Rule Path
Disallow /

hload

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hmview

Rule Path
Disallow /

hometown spider pro

Rule Path
Disallow /

ht://dig

Rule Path
Disallow /

html_analyzer

Rule Path
Disallow /

htmlgobble

Rule Path
Disallow /

httpdownload/2.0

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

hyper-decontextualizer

Rule Path
Disallow /

hð´mð´hð´kki

Rule Path
Disallow /

harvest-ng/1.0.2

Rule Path
Disallow /

haste/0.12 (home: http://haste.kytoon.com/)

Rule Path
Disallow /

hatena antenna/0.4 (http://a.hatena.ne.jp/help/robot)

Rule Path
Disallow /

heinrichdermiragorobot

Rule Path
Disallow /

heinrichdermiragorobot (http://www.miragorobot.com/scripts/deinfo.asp)

Rule Path
Disallow /

henrythemiragorobot

Rule Path
Disallow /

hippias/0.9 beta

Rule Path
Disallow /

hitlist

Rule Path
Disallow /

hitwise spider v1.0 http://www.hitwise.com

Rule Path
Disallow /

hloader

Rule Path
Disallow /

holmes/x.x

Rule Path
Disallow /

homepagesearch(hpsearch.uni-trier.de)

Rule Path
Disallow /

homerbot: www.homerweb.com

Rule Path
Disallow /

hotjava/1.0.1/jre1.1.x

Rule Path
Disallow /

hotzonu/x.0

Rule Path
Disallow /

hoowwwer/2.1.0 (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>hiit.fi)

Rule Path
Disallow /

htdig/3.1.x (root@localhost)

Rule Path
Disallow /

htdig/3.1.6 (http://computerorgs.com)

Rule Path
Disallow /

html link validator (www.lithopssoft.com)

Rule Path
Disallow /

http://ask.24x.info/ (http://narres.it/)

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler [wf216]

Rule Path
Disallow /

httpresume v. 1.x

Rule Path
Disallow /

i, robot

Rule Path
Disallow /

iajabot

Rule Path
Disallow /

ibm_planetwide

Rule Path
Disallow /

icb

Rule Path
Disallow /

image.kapsi.net

Rule Path
Disallow /

imagefetch

Rule Path
Disallow /

imagelock

Rule Path
Disallow /

incywincy

Rule Path
Disallow /

industry

Rule Path
Disallow /

indy

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

informant

Rule Path
Disallow /

infoseek robot 1.0

Rule Path
Disallow /

infoseek sidewinder

Rule Path
Disallow /

infospiders

Rule Path
Disallow /

ingrid

Rule Path
Disallow /

inktomi slurp

Rule Path
Disallow /

inspector web

Rule Path
Disallow /

intelliagent

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet cruiser robot

Rule Path
Disallow /

internet shinchakubin

Rule Path
Disallow /

iron33

Rule Path
Disallow /

iron33/1.0.2

Rule Path
Disallow /

iseeker

Rule Path
Disallow /

israeli-search

Rule Path
Disallow /

iu

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

ia_archiver/1.6

Rule Path
Disallow /

iaarchiver-1.0

Rule Path
Disallow /

ibrowse/2.2 (amigaos 3.5)

Rule Path
Disallow /

ibrowse/2.2 (windows 3.1)

Rule Path
Disallow /

icab/2.5.2 (macintosh; i; ppc)

Rule Path
Disallow /

ice browser/5.05 (java 1.4.0; windows 2000 5.0 x86)

Rule Path
Disallow /

ichiro/1.0 (ichiro@nttr.co.jp)

Rule Path
Disallow /

iconsurf/2.0 favicon finder (see http://iconsurf.com/robot.html)

Rule Path
Disallow /

iconsurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)

Rule Path
Disallow /

icoo loader v.x.x.x

Rule Path
Disallow /

icra_label_spider/x.0

Rule Path
Disallow /

icsbot-0.1

Rule Path
Disallow /

ideare - signsite/1.x

Rule Path
Disallow /

iefav172free

Rule Path
Disallow /

igetter/1.x (macintosh;g;ppc)

Rule Path
Disallow /

iltrovatore-setaccio (+http://www.iltrovatore.it)

Rule Path
Disallow /

iltrovatore-setaccio/0.3-dev (indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)

Rule Path
Disallow /

iltrovatore-setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)

Rule Path
Disallow /

iltrovatore-setaccio/1.2 (it-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)

Rule Path
Disallow /

iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....)

Rule Path
Disallow /

incywincy(http://www.look.com)

Rule Path
Disallow /

incywincy(http://www.loopimprovements.com/robot.html)

Rule Path
Disallow /

incywincy/2.1(loopimprovements.com/robot.html)

Rule Path
Disallow /

incywincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovements.com/robot.html)

Rule Path
Disallow /

incywincy page crawler(webmaster@loopimprovements.com,http://www.loopimprovements.com/robot.html)

Rule Path
Disallow /

indextheweb.com crawler7

Rule Path
Disallow /

industry program 1.0.x

Rule Path
Disallow /

inelabot/0.2 (+http://inelegant.org/bot)

Rule Path
Disallow /

inet library

Rule Path
Disallow /

infofly/1.0 (http://www.versions-project.org/)

Rule Path
Disallow /

infolink/1.x

Rule Path
Disallow /

infomine/8.0 adders

Rule Path
Disallow /

infomine/8.0 remoteservices

Rule Path
Disallow /

infomine/8.0 vlcrawler (http://infomine.ucr.edu/useragents)

Rule Path
Disallow /

infonavirobot(f107)

Rule Path
Disallow /

infoseek sidewinder/0.9

Rule Path
Disallow /

infoseek sidewinder/1.0a

Rule Path
Disallow /

infoseek sidewinder/1.1a

Rule Path
Disallow /

infoseek sidewinder/1.45 (compatible; msie 10.0; unix)

Rule Path
Disallow /

infoseek sidewinder/2.0b (linux 2.4 i686)

Rule Path
Disallow /

inktomi search

Rule Path
Disallow /

innerprisebot/1.0 (http://www.innerprise.com/)

Rule Path
Disallow /

installshield digitalwizard

Rule Path
Disallow /

internetlinkagent/3.1

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

internet ninja 6.0

Rule Path
Disallow /

intravnews/1.x

Rule Path
Disallow /

ip*works! v5 http/s component - by /n software - www.nsoftware.com

Rule Path
Disallow /

ipiumbot laurion(dot)com

Rule Path
Disallow /

iria/1.xxa

Rule Path
Disallow /

irlbot/1.0 (+http://irl.cs.tamu.edu/crawler)

Rule Path
Disallow /

irvine/1.x.x

Rule Path
Disallow /

isurf (tszhu@canada.com)

Rule Path
Disallow /

iupui research bot v 1.9a

Rule Path
Disallow /

java

Rule Path
Disallow /

javabee

Rule Path
Disallow /

jbot java web robot

Rule Path
Disallow /

jcrawler

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jobo java web robot

Rule Path
Disallow /

jobot

Rule Path
Disallow /

joebot

Rule Path
Disallow /

jumpstation

Rule Path
Disallow /

j-phone/3.0/j-sh07

Rule Path
Disallow /

jabot/7.x.x (http://odin.ingrid.org/)

Rule Path
Disallow /

jabot/6.x (http://odin.ingrid.org/)

Rule Path
Disallow /

jack

Rule Path
Disallow /

jakarta commons-httpclient/2.0xxx

Rule Path
Disallow /

java/1.4.1_01

Rule Path
Disallow /

java1.0.21.0

Rule Path
Disallow /

java 1.1

Rule Path
Disallow /

java1.1.xx.x

Rule Path
Disallow /

java1.3.x

Rule Path
Disallow /

java1.3.0rc1

Rule Path
Disallow /

java1.4.0

Rule Path
Disallow /

jbh agent 2.0

Rule Path
Disallow /

jbrowser/j2me profile/midp-1.0 configuration/cldc-1.0 (google wap proxy/1.0)

Rule Path
Disallow /

jdk/1.1

Rule Path
Disallow /

jchecklinks/0.1 rpt-httpclient/0.3-1

Rule Path
Disallow /

jeode/1.x.x

Rule Path
Disallow /

jetbrains omea reader 1.0.x (http://www.jetbrains.com/omea_reader/)

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jetbot/1.0

Rule Path
Disallow /

jigsaw/2.2.0 w3c_css_validator_jfouffa/2.0

Rule Path
Disallow /

jobo/@jobo_version@(http://www.matuschek.net/jobo.html)

Rule Path
Disallow /

jobo/1.x (http://www.matuschek.net/jobo.html)

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

jrts check favorites utility

Rule Path
Disallow /

jrtwine software check favorites utility

Rule Path
Disallow /

jyxobot/x

Rule Path
Disallow /

katipo

Rule Path
Disallow /

kdd-explorer

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

kilroy

Rule Path
Disallow /

kit-fireball

Rule Path
Disallow /

ko_yappo_robot

Rule Path
Disallow /

k-meleon/0.6 (windows; u; windows nt 5.1; en-us; rv:0.9.5) gecko/20011011

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

kapere (http://www.kapere.com)

Rule Path
Disallow /

kddi-sn22 up.browser/6.0.7 (gui) mmp/1.1 (google wap proxy/1.0)

Rule Path
Disallow /

ke_1.0/2.0 libwww/5.2.8

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

kevin http://dznet.com/kevin/

Rule Path
Disallow /

kevin http://websitealert.net/kevin/

Rule Path
Disallow /

kit-fireball/2.0

Rule Path
Disallow /

kit-fireball/2.0 (compatible; mozilla 4.0; msie 5.5)

Rule Path
Disallow /

klondike/1.50 (wsp win32) (google wap proxy/1.0)

Rule Path
Disallow /

knowledge.com/0.x

Rule Path
Disallow /

kontiki client x.xx

Rule Path
Disallow /

kulokobot www.kuloko.com kuloko@backweave.com

Rule Path
Disallow /

kuloko-bot/0.x

Rule Path
Disallow /

kulturarw3/0.1

Rule Path
Disallow /

labelgrabber

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechget

Rule Path
Disallow /

legs

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

libfetch

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

lincoln

Rule Path
Disallow /

link validator

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

links aromatized

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkscan/8.1a unix

Rule Path
Disallow /

lockon

Rule Path
Disallow /

logo.gif crawler

Rule Path
Disallow /

lwp

Rule Path
Disallow /

lwp::simple

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

lachesis

Rule Path
Disallow /

lcabotaccept: */*

Rule Path
Disallow /

lecodechecker/3.0 libgetdoc/1.0

Rule Path
Disallow /

leechget 200x (www.leechget.de)

Rule Path
Disallow /

leia/2.90

Rule Path
Disallow /

leia/3.01pr (leiacrawler; [snip])

Rule Path
Disallow /

lexibot/1.00

Rule Path
Disallow /

lg/u8138/v1.0

Rule Path
Disallow /

libby_1.1/libwww-perl/5.47

Rule Path
Disallow /

libweb/clshttp -- hiongun@kt.co.kr

Rule Path
Disallow /

libwww/5.3.2

Rule Path
Disallow /

libwww-perl/5.41

Rule Path
Disallow /

libwww-perl/5.45

Rule Path
Disallow /

libwww-perl/5.48

Rule Path
Disallow /

libwww-perl/5.50

Rule Path
Disallow /

libwww-perl/5.52 fp/2.1

Rule Path
Disallow /

libwww-perl/5.52 fp/4.0

Rule Path
Disallow /

libwww-perl/5.53

Rule Path
Disallow /

libwww-perl/5.63

Rule Path
Disallow /

libwww-perl/5.64

Rule Path
Disallow /

libwww-perl/5.65

Rule Path
Disallow /

lightningdownload/1.x.x

Rule Path
Disallow /

lightningdownload/1.0beta2

Rule Path
Disallow /

lightningdownload/1.x.x [accelerated x]

Rule Path
Disallow /

lincoln state web browser

Rule Path
Disallow /

linkalarm/2.x

Rule Path
Disallow /

link valet online 1.x

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

linkbot

Rule Path
Disallow /

linkbot x.0

Rule Path
Disallow /

linkcheck (linkcheck@inter7.com http://www.inter7.com/linkcheck)

Rule Path
Disallow /

linklint-checkonly/2.x.x

Rule Path
Disallow /

linklint-spider/2.x.x

Rule Path
Disallow /

linkprover 2.1

Rule Path
Disallow /

links (0.9x; linux 2.4.7-10 i686)

Rule Path
Disallow /

links (0.9xpre12; linux 2.2.14-5.0 i686; 80x24)

Rule Path
Disallow /

links (2.xpre7; linux 2.4.18 i586; x)

Rule Path
Disallow /

links - http://gossamer-threads.com/scripts/links/

Rule Path
Disallow /

links 2.0 (http://gossamer-threads.com/scripts/links/)

Rule Path
Disallow /

links sql (http://gossamer-threads.com/scripts/links-sql/)

Rule Path
Disallow /

linkscan/x.x unix

Rule Path
Disallow /

linkscan/9.0g unix

Rule Path
Disallow /

linkscan/11.0beta2 unix

Rule Path
Disallow /

linksmanager.com (http://linksmanager.com/linkchecker.html)

Rule Path
Disallow /

linksweeper/1.x

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

linknzbot

Rule Path
Disallow /

listbidbot (freelance job spider http://listbid.com)<a href=http://listbid.com>freelance</a>

Rule Path
Disallow /

llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)

Rule Path
Disallow /

lmspider (lmspider@scansoft.com)

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

lockstep spider/1.0

Rule Path
Disallow /

look.com

Rule Path
Disallow /

lotus-notes/4.5 ( windows-nt )

Rule Path
Disallow /

lotusdiscovery/x.0 (compatible; mozilla 4.0; msie 4.01; windows nt)

Rule Path
Disallow /

lwp::simple/5.22

Rule Path
Disallow /

lwp::simple/5.36

Rule Path
Disallow /

lwp::simple/5.48

Rule Path
Disallow /

lwp::simple/5.50

Rule Path
Disallow /

lwp::simple/5.51

Rule Path
Disallow /

lwp::simple/5.53

Rule Path
Disallow /

lwp::simple/5.63

Rule Path
Disallow /

lwp-trivial/1.32

Rule Path
Disallow /

lwp-trivial/1.34

Rule Path
Disallow /

lwp-trivial/1.35

Rule Path
Disallow /

lycos

Rule Path
Disallow

t-rex (lycos)

Rule Path
Disallow

lycos_spider_(t-rex)

Rule Path
Disallow

lycos_spider_(modspider)

Rule Path
Disallow /

lynx/2.6 libwww-fm/2.14

Rule Path
Disallow /

lynx/2.8.3dev.9 libwww-fm/2.14 ssl-mm/1.4.1 openssl/0.9.6

Rule Path
Disallow /

lynx/2.8.4rel.1 libwww-fm/2.14 ssl-mm/1.4.1 openssl/0.9.6c (human-guided@lerly.net)

Rule Path
Disallow /

mmcrawler

Rule Path
Disallow /

mac finder

Rule Path
Disallow /

mac wwwworm

Rule Path
Disallow /

magpie

Rule Path
Disallow /

marvin/infoseek

Rule Path
Disallow /

mass

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mattie

Rule Path
Disallow /

md

Rule Path
Disallow /

mediafox

Rule Path
Disallow /

mega moose

Rule Path
Disallow /

memoweb

Rule Path
Disallow /

merzscope

Rule Path
Disallow /

mfc_tear

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mindcrawler

Rule Path
Disallow /

mirago

Rule Path
Disallow /

mirror

Rule Path
Disallow /

missauga

Rule Path
Disallow /

missigua

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

miz

Rule Path
Disallow /

mm3

Rule Path
Disallow /

mnogo

Rule Path
Disallow /

mnogosearch search engine software

Rule Path
Disallow /

moget

Rule Path
Disallow /

moget/2.1

Rule Path
Disallow /

momspider

Rule Path
Disallow /

monster

Rule Path
Disallow /

mooz

Rule Path
Disallow /

motor

Rule Path
Disallow /

muncher

Rule Path
Disallow /

muninn

Rule Path
Disallow /

muscat ferret

Rule Path
Disallow /

mwd.search

Rule Path
Disallow /

mygetright

Rule Path
Disallow /

mac finder 1.0.xx

Rule Path
Disallow /

mackster( http://www.ukwizz.com )

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

mailto:webcraft@bea.com

Rule Path
Disallow /

mammoth/1.0 (+http://www.sli-systems.com/)

Rule Path
Disallow /

mantraagent

Rule Path
Disallow /

mariner/5.1b [de] (win95; i ;kolibri gncwebbot)

Rule Path
Disallow /

marketwave hit list

Rule Path
Disallow /

martini

Rule Path
Disallow /

marvin v0.3

Rule Path
Disallow /

mass downloader 2.x

Rule Path
Disallow /

mata hari/2.00

Rule Path
Disallow /

maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/maxomo/maxomobot.html; maxomobot@maxomo.com)

Rule Path
Disallow /

mediacrawler-1.0 (experimental)

Rule Path
Disallow /

mediapartners-google/2.1 (+http://www.googlebot.com/bot.html)

Rule Path
Disallow

mediasearch/0.1

Rule Path
Disallow /

megasheep v1.0 (www.searchuk.com internet sheep)

Rule Path
Disallow /

mercator-1.x

Rule Path
Disallow /

mercator-2.0

Rule Path
Disallow /

mercator-scrub-1.1

Rule Path
Disallow /

metager-linkchecker

Rule Path
Disallow /

metaproducts download express/1.x

Rule Path
Disallow /

metaspinner/0.01 (metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)

Rule Path
Disallow /

mfc foundation class library 4.0

Rule Path
Disallow /

mfc_tear_sample

Rule Path
Disallow /

mfhttpscan

Rule Path
Disallow /

microsoft data access internet publishing provider protocol discovery

Rule Path
Disallow /

microsoftprototypecrawler (how's my crawling? mailto:newbiecrawler@hotmail.com)

Rule Path
Disallow /

microsoft url control - 6.00.8xxx

Rule Path
Disallow

miixpc/4.2

Rule Path
Disallow /

minibot

Rule Path
Disallow /

miraclealphatest

Rule Path
Disallow /

missauga locate 1.0.0

Rule Path
Disallow /

missigua locator 1.9

Rule Path
Disallow /

missouri college browse

Rule Path
Disallow /

mister pix version.dll

Rule Path
Disallow /

mister pix ii 2.02a

Rule Path
Disallow /

mj12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)

Rule Path
Disallow /

mnogosearch/3.2.xx

Rule Path
Disallow /

mouse-house/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml)

Rule Path
Disallow /

moget/x.x (moget@goo.ne.jp)

Rule Path
Disallow /

mogimogi/1.0

Rule Path
Disallow /

moinag 0.02

Rule Path
Disallow /

mojeekbot/0.x (archi; http://www.mojeek.com/bot.html)

Rule Path
Disallow /

morris - mixcat crawler (+http://mixcat.com)

Rule Path
Disallow /

movabletype/x.x

Rule Path
Disallow /

mozdex/0.04-dev (mozdex; http://www.mozdex.com/en/bot.html; spider@mozdex.com)

Rule Path
Disallow /

mozi!

Rule Path
Disallow /

msproxy/2.0

Rule Path
Disallow /

msnptc/1.0

Rule Path
Disallow /

multiblocker browser

Rule Path
Disallow /

multitext/0.1

Rule Path
Disallow /

musicwalker2.0 (+http://www.somusical.com)

Rule Path
Disallow /

mydaemon

Rule Path
Disallow /

mygetright/1.0b

Rule Path
Disallow /

mygetright/1.0.0

Rule Path
Disallow /

mylinea.com crawler 2.0

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

navroad

Rule Path
Disallow /

navroad

Rule Path
Disallow /

ndspider

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

nec-meshexplorer

Rule Path
Disallow /

nederland.zoek

Rule Path
Disallow /

net m@nager

Rule Path
Disallow /

net\ vampire

Rule Path
Disallow /

netants

Rule Path
Disallow /

netcarta webmap engine

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netscoop

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netstreamrecorder

Rule Path
Disallow /

newscan-online

Rule Path
Disallow /

nhse web forager

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nomad

Rule Path
Disallow /

northern light gulliver

Rule Path
Disallow /

np

Rule Path
Disallow /

nzexplorer

Rule Path
Disallow /

nabot_1.0

Rule Path
Disallow /

nabot/5.0

Rule Path
Disallow /

nationaldirectoryaddurl/1.0

Rule Path
Disallow /

nationaldirectory-webspider/1.3

Rule Path
Disallow /

naverbot-1.0 (nhn corp. / +82-2-3011-1954 / nhnbot@naver.com)

Rule Path
Disallow /

naverbot_dloader/1.5

Rule Path
Disallow /

ncsa beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)

Rule Path
Disallow /

nec-hayek/1.0

Rule Path
Disallow /

nec research agent -- compuman at research.nj.nec.com

Rule Path
Disallow /

netants/1.2x

Rule Path
Disallow /

netcomplete/x.xx

Rule Path
Disallow /

netlookout/2.24

Rule Path
Disallow /

netmechanic vx.0

Rule Path
Disallow /

netnosecrawler/v1.0

Rule Path
Disallow /

netprospector javacrawler

Rule Path
Disallow /

netpumper/x.xx

Rule Path
Disallow /

netresearchserver(http://www.look.com)

Rule Path
Disallow /

netresearchserver/x.x(loopimprovements.com/robot.html)

Rule Path
Disallow /

netzippy

Rule Path
Disallow /

neuralbot/0.2

Rule Path
Disallow /

nextgensearchbot 1 (for information visit http://www.eliyon.com/nextgensearchbot)

Rule Path
Disallow /

nextopiabot (+http://www.nextopia.com) distributed crawler client beta v0.x

Rule Path
Disallow /

newsearchengine (thisuser@unspecified.mail)

Rule Path
Disallow /

ng/1.0

Rule Path
Disallow /

nico/1.0

Rule Path
Disallow /

nitle blog spider/0.01

Rule Path
Disallow /

nitro downloader 1.x (www.klsofttools.com)

Rule Path
Disallow /

noago spider

Rule Path
Disallow /

nocilla/1.0

Rule Path
Disallow /

nokia-waptoolkit/1.2 googlebot(at)googlebot.com

Rule Path
Disallow

nokia7110/1.0 (05.01) (google wap proxy/1.0)

Rule Path
Disallow

nokodobot/1.x (+http://nokodo.com/bot.htm)

Rule Path
Disallow /

np/0.1 (np; http://www.nameprotect.com; npbot@nameprotect.com)

Rule Path
Disallow /

npbot (http://www.nameprotect.com/botinfo.html)

Rule Path
Disallow /

npbot-1/2.0

Rule Path
Disallow /

nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp)

Rule Path
Disallow /

nusearch spider <a href='http://www.nusearch.com'>www.nusearch.com</a> (compatible; msie 4.01)

Rule Path
Disallow /

nusearch spider www.nusearch.com

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nutchcvs/0.0x-dev (nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)

Rule Path
Disallow /

nutchorg/0.0x-dev (nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

obot

Rule Path
Disallow /

occam

Rule Path
Disallow /

offline

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

offline\ navigator

Rule Path
Disallow /

ontospider

Rule Path
Disallow /

openbot

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openfind data gathere

Rule Path
Disallow /

openfind data gatherer

Rule Path
Disallow /

orb search

Rule Path
Disallow /

obidos-bot (just looking for books.)

Rule Path
Disallow /

objectssearch/0.01-dev (objectssearch;http://www.objectssearch.com/bot.html; support@thesoftwareobjects.com)

Rule Path
Disallow /

obot ((compatible;win32))

Rule Path
Disallow /

ocelli/1.x (http://www.globalspec.com/ocelli)

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline explorer 1.*

Rule Path
Disallow /

oliverperry

Rule Path
Disallow /

omniexplorer_bot/1.07 (+http://www.omni-explorer.com) internet categorizer

Rule Path
Disallow /

omniexplorer_bot/1.07 (+http://www.omni-explorer.com) job crawler

Rule Path
Disallow /

omniexplorer_bot/1.18 (+http://www.omni-explorer.com) torrent crawler

Rule Path
Disallow /

online link validator (http://www.dead-links.com/)

Rule Path
Disallow /

ontospider/1.0 libwww-perl/5.65

Rule Path
Disallow /

openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)

Rule Path
Disallow /

openfind data gatherer, openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)

Rule Path
Disallow /

openfind robot/1.1a2

Rule Path
Disallow /

opentextsitecrawler/2.9.2

Rule Path
Disallow /

openwebspider/x

Rule Path
Disallow /

oracle imtcrawler

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

overture-webcrawler/3.8/fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

Rule Path
Disallow /

owr_crawler 0.1

Rule Path
Disallow /

pack rat

Rule Path
Disallow /

pageboy

Rule Path
Disallow /

pagedown

Rule Path
Disallow /

pageripper

Rule Path
Disallow /

pagesucker

Rule Path
Disallow /

parasite

Rule Path
Disallow /

patric

Rule Path
Disallow /

patwebbot

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pegasus

Rule Path
Disallow /

perlcrawler 1.0

Rule Path
Disallow /

personal internet engine

Rule Path
Disallow /

personapilot

Rule Path
Disallow /

pgp key agent

Rule Path
Disallow /

phantom

Rule Path
Disallow /

phpdig

Rule Path
Disallow /

piltdownman

Rule Path
Disallow /

pimptrain.com's robot

Rule Path
Disallow /

pioneer

Rule Path
Disallow /

plumtreewebaccessor

Rule Path
Disallow /

poppi

Rule Path
Disallow /

popular iconoclast

Rule Path
Disallow /

portal juice spider

Rule Path
Disallow /

portalb spider

Rule Path
Disallow /

program

Rule Path
Disallow /

project

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

pagmiedownload

Rule Path
Disallow /

parallelcontextfocuscrawler1.1parallelcontextfocuscrawler1.1

Rule Path
Disallow /

parasite/1.0b (http://www.ianett.com/parasite/)

Rule Path
Disallow /

patwebbot (http://www.herz-power.de/technik.html)

Rule Path
Disallow /

pavuk/0.9pl29b i686-pc-linux-gnu

Rule Path
Disallow /

pbrowse 1.4b

Rule Path
Disallow /

pd02_1.0.0 pd02_1.0.0@dzimi@post.sk

Rule Path
Disallow /

pear http_request class ( http://pear.php.net/ )

Rule Path
Disallow /

peerbot www.peerbot.com

Rule Path
Disallow /

peoplechat/search_engine

Rule Path
Disallow /

peval 1.4b

Rule Path
Disallow /

php/3.x.xx

Rule Path
Disallow /

php/4.0.4pl1

Rule Path
Disallow /

php/4.0.6

Rule Path
Disallow /

php/4.1.1

Rule Path
Disallow /

php/4.1.2

Rule Path
Disallow /

picosearch/1.0

Rule Path
Disallow /

pingalink monitoring services 1.0

Rule Path
Disallow /

pingalink monitoring services 1.0 (http://www.pingalink.com)

Rule Path
Disallow /

pipeliner/0.10 (pipeline spider; http://www.pipeline-search.com/webmaster.html)

Rule Path
Disallow /

pipeliner/0.3a (pipeline spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com)

Rule Path
Disallow /

pita

Rule Path
Disallow /

pizilla++ ver 2.45

Rule Path
Disallow /

pjspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)

Rule Path
Disallow /

plagiarbot/1.0

Rule Path
Disallow /

plantynet_webrobot_v1.9 dhkang@plantynet.com

Rule Path
Disallow /

pockey/x.x.x

Rule Path
Disallow /

pockey7.x.x(win32gui)

Rule Path
Disallow /

pockey-gethtml/x.xx

Rule Path
Disallow /

pockey-gethtml/4.12.0 (win32; gui; ix86)

Rule Path
Disallow /

poe-component-client-http/0.64 (perl; n; poe; en; rv:0.640000)

Rule Path
Disallow /

polybot 1.0 (http://cis.poly.edu/polybot/)

Rule Path
Disallow /

pompos/1.x pompos@iliad.fr

Rule Path
Disallow /

pompos/1.x http://dir.com/pompos.html

Rule Path
Disallow /

popdexter/1.0

Rule Path
Disallow /

portalbspider/2.0 (spider@portalb.com)

Rule Path
Disallow /

potbot 1.0

Rule Path
Disallow /

privoxy/3.0 (anonymous)

Rule Path
Disallow /

production bot 0116b

Rule Path
Disallow /

production bot 2016b

Rule Path
Disallow /

production bot dot 3016b

Rule Path
Disallow /

program shareware 1.0.2

Rule Path
Disallow /

progressive download

Rule Path
Disallow /

progressive download http check

Rule Path
Disallow /

project xp5 [2.03.07-111203]

Rule Path
Disallow /

prove answerbot 4.0

Rule Path
Disallow /

prowebguide link checker (http://www.prowebguide.com)

Rule Path
Disallow /

psbot/0.1 (+http://www.picsearch.com/bot.html)

Rule Path
Disallow /

psurf15a 11

Rule Path
Disallow /

psurf15a 51

Rule Path
Disallow /

psurf15a va

Rule Path
Disallow /

pubcrawl (pubcrawl.stanford.edu)

Rule Path
Disallow /

puf/0.91beta6a (linux 2.2.18; i686)

Rule Path
Disallow /

puresight

Rule Path
Disallow /

puxarapido v1.0

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python-urllib/1.15

Rule Path
Disallow /

python-urllib/2.0a1

Rule Path
Disallow /

que

Rule Path
Disallow /

qango.com web directory (http://www.qango.com/)

Rule Path
Disallow /

qpcreep test rig ( we are not indexing, just testing )

Rule Path
Disallow /

quepasacreep ( crawler@quepasacorp.com )

Rule Path
Disallow /

quepasacreep v0.9.1x

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

radiation retriever 1.1

Rule Path
Disallow /

rafabot

Rule Path
Disallow /

raven search

Rule Path
Disallow /

rbse spider

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

resume robot

Rule Path
Disallow /

river

Rule Path
Disallow /

rixbot

Rule Path
Disallow /

rma

Rule Path
Disallow /

road runner: the imagescape robot

Rule Path
Disallow /

roadhouse crawling system

Rule Path
Disallow /

roadrunner

Rule Path
Disallow /

robbie the robot

Rule Path
Disallow /

robocrawl spider

Rule Path
Disallow /

robofox

Rule Path
Disallow /

robot francoroute

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

rpt

Rule Path
Disallow /

rules

Rule Path
Disallow /

rabaz (rabaz at gigabaz dot com)

Rule Path
Disallow /

rabot/1.0 agent-admin/phortse@hanmail.net

Rule Path
Disallow /

rainbot1.1

Rule Path
Disallow /

rambot xtreme x.x

Rule Path
Disallow /

rational sitecheck (windows nt)

Rule Path
Disallow /

realdownload/4.0.0.4x

Rule Path
Disallow /

reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)

Rule Path
Disallow /

reaper/2.0x (+http://www.sitesearch.ca/reaper)

Rule Path
Disallow /

rebol core 2.x.x.x.x

Rule Path
Disallow /

rel link checker lite x.x

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

rewebber/1.2 libwww-perl/5.41

Rule Path
Disallow /

rico/0.1

Rule Path
Disallow /

rma/1.0 (compatible; realmedia)

Rule Path
Disallow /

robocrawl (www.canadiancontent.net)

Rule Path
Disallow /

robopal (http://www.findpal.com/)

Rule Path
Disallow /

robot@supersnooper.com

Rule Path
Disallow /

robot/www.pj-search.com

Rule Path
Disallow /

robot: nutchcrawler, owner: wdavies@acm.org

Rule Path
Disallow /

robozilla/1.0

Rule Path
Disallow /

rotondo/3.1 libwww/5.3.1

Rule Path
Disallow /

rpt-httpclient/0.3-x

Rule Path
Disallow /

rrc (crawler_admin@bigfoot.com)

Rule Path
Disallow /

rsurf15a 41

Rule Path
Disallow /

rsurf15a 51

Rule Path
Disallow /

rsurf15a 81

Rule Path
Disallow /

rumours-agent

Rule Path
Disallow /

safetynet robot

Rule Path
Disallow /

scooter

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

search.aus-au.com

Rule Path
Disallow /

searchprocess

Rule Path
Disallow /

secret

Rule Path
Disallow /

selfseek

Rule Path
Disallow /

senrigan

Rule Path
Disallow /

sg-scout

Rule Path
Disallow /

shagseeker

Rule Path
Disallow /

shai'hulud

Rule Path
Disallow /

sift

Rule Path
Disallow /

simmany robot ver1.0

Rule Path
Disallow /

simple/

Rule Path
Disallow /

site searcher

Rule Path
Disallow /

site valet

Rule Path
Disallow /

sitemapper

Rule Path
Disallow /

sitetech-rover

Rule Path
Disallow /

skymob.com

Rule Path
Disallow /

slcrawler

Rule Path
Disallow /

sleek

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smart spider

Rule Path
Disallow /

snapitup

Rule Path
Disallow /

snoop

Rule Path
Disallow /

snooper

Rule Path
Disallow /

solbot

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

spider_monkey

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

spiderline crawler

Rule Path
Disallow /

spiderview(tm)

Rule Path
Disallow /

spry wizard robot

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

star

Rule Path
Disallow /

suke

Rule Path
Disallow /

sundoh

Rule Path
Disallow /

suntek search engine

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surfsaver

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

sven

Rule Path
Disallow /

sygol

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

s.t.a.l.k.e.r. (http://www.seo-tools.net/en/bot.aspx)

Rule Path
Disallow /

safaribookmarkchecker (+http://www.coriolis.ch/)

Rule Path
Disallow /

sbl-bot (http://sbl.net)

Rule Path
Disallow /

scanweb

Rule Path
Disallow /

schwarzmann.biz-spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp)

Rule Path
Disallow /

science traveller international 1x/1.0

Rule Path
Disallow /

scooter/1.0

Rule Path
Disallow /

scooter/1.0 scooter@pa.dec.com

Rule Path
Disallow /

scooter/1.1 (custom)

Rule Path
Disallow /

scooter/2.0 g.r.a.b. v1.1.0

Rule Path
Disallow /

scooter/2.0 g.r.a.b. x2.0

Rule Path
Disallow /

scooter-3.0.eu

Rule Path
Disallow /

scooter-3.0.fs

Rule Path
Disallow /

scooter-3.0.hd

Rule Path
Disallow /

scooter-3.0.vns

Rule Path
Disallow /

scooter-3.0qi

Rule Path
Disallow /

scooter-3.2

Rule Path
Disallow /

scooter-3.2.bt

Rule Path
Disallow /

scooter-3.2.dil

Rule Path
Disallow /

scooter-3.2.ex

Rule Path
Disallow /

scooter-3.2.jt

Rule Path
Disallow /

scooter-3.2.niv

Rule Path
Disallow /

scooter-3.2.sf0

Rule Path
Disallow /

scooter-3.2.snippet

Rule Path
Disallow /

scooter/3.3

Rule Path
Disallow /

scooter/3.3.qa.pczukor

Rule Path
Disallow /

scooter/3.3_sf

Rule Path
Disallow /

scooter/3.3.vscooter

Rule Path
Disallow /

scooter-3.3dev

Rule Path
Disallow /

scooter-ars-1.1

Rule Path
Disallow /

scooter-ars-1.1-ih

Rule Path
Disallow /

scooter_bh0-3.0.3

Rule Path
Disallow /

scooter_trk3-3.0.3

Rule Path
Disallow /

scooter-venus-3.0.vns

Rule Path
Disallow /

scooter-w3-1.0

Rule Path
Disallow /

scooter-w3.1.2

Rule Path
Disallow /

scooter2_mercator_x-x.0

Rule Path
Disallow /

scoutabout

Rule Path
Disallow /

scrubby/2.x (http://www.scrubtheweb.com/)

Rule Path
Disallow /

search+

Rule Path
Disallow /

search/1.0 (http://www.innerprise.net/es-spider.asp)

Rule Path
Disallow /

search.ch v1.4

Rule Path
Disallow /

search.ch v1.4.2 (spiderman@search.ch; http://www.search.ch)

Rule Path
Disallow /

searchbyusa/2 (searchbyusa; http://www.searchbyusa.com/bot.html; info@searchbyusa.com)

Rule Path
Disallow /

searchexpress spider0.99

Rule Path
Disallow /

searchguild_dmoz_experiment (chris@searchguild.com)

Rule Path
Disallow /

searchit-now robot/2.2 (+http://www.searchit-now.co.uk)

Rule Path
Disallow /

searchspider.com/1.1

Rule Path
Disallow /

searchspider/1.2 (searchspider; http://www.searchspider.com; webmaster@searchspider.com)

Rule Path
Disallow /

searchtone2.0 - ideare

Rule Path
Disallow /

seeker.lookseek.com

Rule Path
Disallow /

seekbot/1.0 (http://www.seekbot.net/bot.html) httpfetcher/0.3

Rule Path
Disallow /

seekbot/1.0 (http://www.seekbot.net/bot.html) robotstxtfetcher/1.0 (xdf)

Rule Path
Disallow /

semanticdiscovery/0.x

Rule Path
Disallow /

sensis.com.au web crawler (search_comments\at\sensis\dot\com\dot\au)

Rule Path
Disallow /

seznambot/1.0

Rule Path
Disallow /

seznambot/1.0 (+http://fulltext.seznam.cz/)

Rule Path
Disallow /

shareaza v1.x.x.xx

Rule Path
Disallow /

sharewareplazafilecheckbot/1.0+(+http://www.sharewareplaza.com)

Rule Path
Disallow /

simpy 1.x; http://www.simpy.com/

Rule Path
Disallow /

simpy/1.x (simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)

Rule Path
Disallow /

sitebar/x.x

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitecheck.internetseer.com (for more info see: http://sitecheck.internetseer.com)

Rule Path
Disallow /

siterecon+(xx)

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sitespider +(http://www.sitespider.com/)

Rule Path
Disallow /

sitewinder

Rule Path
Disallow /

sitexpert

Rule Path
Disallow /

skampy/0.9.x (http://www.skaffe.com/skampy-info.html)

Rule Path
Disallow /

skimpy/0.x (http://www.skaffe.com/skampy-info.html)

Rule Path
Disallow /

slarp/0.1

Rule Path
Disallow /

sleipnir version 1.xx

Rule Path
Disallow /

slider_search_v1-de

Rule Path
Disallow /

slimbrowser

Rule Path
Disallow /

slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slurp/2.0-kiteweekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Rule Path
Disallow /

slysearch (slysearch@slysearch.com)

Rule Path
Disallow /

slysearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html

Rule Path
Disallow /

slysearch/1.x http://www.slysearch.com

Rule Path
Disallow /

smartdownload/1.2.67 (win32; jan 12 1999)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; feb 1 2000)

Rule Path
Disallow /

smartdownload/1.2.77 (win32; jun 19 2001)

Rule Path
Disallow /

smartwit.com

Rule Path
Disallow /

sna-0.0.1 mikeelliott@hotmail.com

Rule Path
Disallow /

snooprob/x.x

Rule Path
Disallow /

snoopy_v0.xx

Rule Path
Disallow /

snoopy v1.xx

Rule Path
Disallow /

snoopy v1.xx, : user-agent: mozilla/4.0 (compatible; msie 6.0; windows nt 5.1; myie2)

Rule Path
Disallow /

snykebot/0.6 (http://www.snyke.com)

Rule Path
Disallow /

softhypermarketfilecheckbot/1.0+(+http://www.softhypermaket.com)

Rule Path
Disallow /

sohu-search

Rule Path
Disallow /

speeddownload/1.x

Rule Path
Disallow /

speedfind rambot xtreme 8.1

Rule Path
Disallow /

speedy spider (beta/x.x; speedy@entireweb.com)

Rule Path
Disallow /

speedy_spider (http://www.entireweb.com)

Rule Path
Disallow /

spida/0.1

Rule Path
Disallow /

spider/maxbot.com admin@maxbot.com

Rule Path
Disallow /

spider.batsch.com

Rule Path
Disallow /

spider.yellopet.com - www.yellopet.com

Rule Path
Disallow /

spider-sleek/2.0 (+http://search-info.com/linktous.html)

Rule Path
Disallow /

spiderku/0.x

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

spidermonkey/7.0x (spidermonkey.ca info at http://spidermonkey.ca/sm.shtml)

Rule Path
Disallow /

spinne/2.0

Rule Path
Disallow /

spinne/2.0 med

Rule Path
Disallow /

spinne/2.0 med_ah

Rule Path
Disallow /

sq webscanner

Rule Path
Disallow /

sqworm/2.9.81-beta (beta_release; 20011102-760; i686-pc-linux-gnu)

Rule Path
Disallow /

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux-gnu)

Rule Path
Disallow /

sqworm/2.9.89-beta (beta_release; 20020130-839; i686-pc-linux-gnu)

Rule Path
Disallow /

ssurf15a 11

Rule Path
Disallow /

stackrambler/x.x

Rule Path
Disallow /

stamina/1.4

Rule Path
Disallow /

star downloader

Rule Path
Disallow /

stat statcrawler@gmail.com

Rule Path
Disallow /

steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

Rule Path
Disallow /

submission spider at surfsafely.com

Rule Path
Disallow /

suchbot

Rule Path
Disallow /

suchknecht.at-robot

Rule Path
Disallow /

superbot/x.x (win32)

Rule Path
Disallow /

surf

Rule Path
Disallow /

surferf3 1/0

Rule Path
Disallow /

surfmaster

Rule Path
Disallow /

surveybot/2.2 <a href='http://www.whois.sc'>whois source</a>

Rule Path
Disallow /

surveybot/2.3 (whois source)

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

swb/v1.4 (hp)

Rule Path
Disallow /

swbot/0.9c libwww/5.3.1

Rule Path
Disallow /

sylera/1.2.x

Rule Path
Disallow /

syncbot

Rule Path
Disallow /

syncit/x.x

Rule Path
Disallow /

synobot

Rule Path
Disallow /

szukacz/1.x

Rule Path
Disallow /

szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)

Rule Path
Disallow /

tach black widow

Rule Path
Disallow /

takeout

Rule Path
Disallow /

tarantula

Rule Path
Disallow /

tarspider

Rule Path
Disallow /

tcl w3 robot

Rule Path
Disallow /

techbot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

teleport\ pro

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

templeton

Rule Path
Disallow /

teoma

Rule Path
Disallow /

terrapin

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

the northstar robot

Rule Path
Disallow /

the nwi robot

Rule Path
Disallow /

the peregrinator

Rule Path
Disallow /

the python robot

Rule Path
Disallow /

the tkwww robot

Rule Path
Disallow /

the web moose

Rule Path
Disallow /

the web wombat

Rule Path
Disallow /

the webfoot robot

Rule Path
Disallow /

the world wide web wanderer

Rule Path
Disallow /

the world wide web worm

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

titan

Rule Path
Disallow /

titan

Rule Path
Disallow /

titin

Rule Path
Disallow /

tlspider

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

true_robot/1.0

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turn it in

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tagword (http://tagword.com/dmoz_survey.php)

Rule Path
Disallow /

talkro web-shot/1.0 (e-mail: webshot@daumsoft.com, home: http://222.122.15.190/webshot)

Rule Path
Disallow /

tamu_cs_irl_crawler/1.0

Rule Path
Disallow /

targetblaster.com/0.9k

Rule Path
Disallow /

te

Rule Path
Disallow /

tecomac-crawler/0.x

Rule Path
Disallow /

teamsoft wininet component

Rule Path
Disallow /

teleport pro/1.2x(.1xxx)

Rule Path
Disallow /

teoma mp

Rule Path
Disallow /

teomaagent crawler-admin@teoma.com

Rule Path
Disallow /

teomaagent1 [crawler-admin@teoma.com]

Rule Path
Disallow /

teoma_agent1

Rule Path
Disallow /

teradex mapper; mapper@teradex.com; http://www.teradex.com

Rule Path
Disallow /

test spider

Rule Path
Disallow /

the expert html source viewer (http://www.expert-html.com)

Rule Path
Disallow /

thesubot/0.1 (www.thesubot.de)

Rule Path
Disallow /

tivraspider/1.0 (crawler@tivra.com)

Rule Path
Disallow /

tjg/spider

Rule Path
Disallow /

tjvmultihttpgrabber component

Rule Path
Disallow /

tkensaku/x.x(http://www.tkensaku.com/q.html)

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

toutatis x.x-x

Rule Path
Disallow /

toutatis x.x (hoppa.com)

Rule Path
Disallow /

toutatis x-xx.x (hoppa.com)

Rule Path
Disallow /

trampelpfad-spider

Rule Path
Disallow /

trampelpfad-spider-v0.1

Rule Path
Disallow /

tricosmetacheck 1.2216-08-1999 (http://www.tricos.com/metacheck)

Rule Path
Disallow /

tsurf15a 11

Rule Path
Disallow /

tulipchain/5.x (http://ostermiller.org/tulipchain/) java/1.x.1_0x (http://java.sun.com/) linux/2.4.17

Rule Path
Disallow /

tulipchain/5.xx (http://ostermiller.org/tulipchain/) java/1.x.1_0x (http://apple.com/) mac_os_x/10.2.8

Rule Path
Disallow /

turnitinbot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)

Rule Path
Disallow /

turnpike emporium linkchecker/0.1

Rule Path
Disallow /

tutorgig/1.5 (+http://www.tutorgig.com/crawler)

Rule Path
Disallow /

tutorial crawler 1.4 (http://www.tutorgig.com/crawler)

Rule Path
Disallow /

twotrees reactive filter v2.0

Rule Path
Disallow /

tygobot

Rule Path
Disallow /

tygoprowler

Rule Path
Disallow /

unchaos (from chaos to order, hybrid web search engine. (vadim_gonchar@unchaos.com))

Rule Path
Disallow /

unchaos bot (hybrid web search engine.(vadim_gonchar@unchaos.com))

Rule Path
Disallow /

unchaosbot (from chaos to order, unchaos hybrid web search engine at www.unchaos.com (info@unchaos.com))

Rule Path
Disallow /

ucsd crawl

Rule Path
Disallow /

udmsearch

Rule Path
Disallow /

unnamed

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

url check

Rule Path
Disallow /

url control

Rule Path
Disallow /

url spider pro

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

ucmore

Rule Path
Disallow /

ucmore crawler app

Rule Path
Disallow /

udm

Rule Path
Disallow /

udmsearch/3.1.x

Rule Path
Disallow /

uiowacrawler/1.0

Rule Path
Disallow /

ultraseek

Rule Path
Disallow /

under the rainbow 2.2

Rule Path
Disallow /

up.browser/3.01-ig01 up.link/3.2.3.4

Rule Path
Disallow /

updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)

Rule Path
Disallow /

upg1 up/4.0 (compatible; blazer 1.0)

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

uptimebot(www.uptimebot.com)

Rule Path
Disallow /

urlblaze

Rule Path
Disallow /

urlgetfile

Rule Path
Disallow /

url spider pro/x.xx (innerprise.net)

Rule Path
Disallow /

url_spider_pro/x.x

Rule Path
Disallow /

url_spider_pro/x.x+(http://www.innerprise.net/usp-spider.asp)

Rule Path
Disallow /

user-agent=mozilla/3.01gold

Rule Path
Disallow /

user-agent: fileheap! file downloader (http://www.fileheap.com)

Rule Path
Disallow /

user-agent: mozilla/4.0 (skizzle! distributed internet spider v1.0 - www.skizzle.com)

Rule Path
Disallow /

usyd-nlp-spider (http://www.it.usyd.edu.au/~vinci/bot.html)

Rule Path
Disallow /

utilmind httpget

Rule Path
Disallow /

utopia webwasher 3.0

Rule Path
Disallow /

valkyrie

Rule Path
Disallow /

vayala

Rule Path
Disallow /

vci

Rule Path
Disallow /

vci webviewer vci webviewer win32

Rule Path
Disallow /

verticrawl

Rule Path
Disallow /

victoria

Rule Path
Disallow /

vision-search

Rule Path
Disallow /

visual web task

Rule Path
Disallow /

void-bot

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voyager

Rule Path
Disallow /

vwbot

Rule Path
Disallow /

vagabondo/1.x mt (webagent@wise-guys.nl)

Rule Path
Disallow /

vagabondo/2.0 mt

Rule Path
Disallow /

vagabondo/2.0 mt (webagent at wise-guys dot nl)

Rule Path
Disallow /

vagabondo/2.0 mt (webagent@nospamwise-guys.nl)

Rule Path
Disallow /

vakes/0.01 (vakes; http://www.vakes.com/; search@vakes.com)

Rule Path
Disallow /

vayala|creep-v0.0.1 (codepoet@wildties.com)

Rule Path
Disallow /

vayalacreep-v0.0.1 (haploid@haploid.com)

Rule Path
Disallow /

vb wininet

Rule Path
Disallow /

versus 0.2 (+http://versus.integis.ch)

Rule Path
Disallow /

verygoodsearch.com.daddylonglegs

Rule Path
Disallow /

verzamelgids.nl - networking4all bot/x.x

Rule Path
Disallow /

vspider

Rule Path
Disallow /

vspider/3.x

Rule Path
Disallow /

w@pspider by wap4.com

Rule Path
Disallow /

w3m2

Rule Path
Disallow /

w3mir

Rule Path
Disallow /

walhello appie

Rule Path
Disallow /

wallpaper (alias crawlpaper)

Rule Path
Disallow /

web

Rule Path
Disallow /

web core / roots

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

web image

Rule Path
Disallow /

web page finder

Rule Path
Disallow /

web sucker

Rule Path
Disallow /

web2map

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webbandit web spider

Rule Path
Disallow /

webbandit/3.50

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

webcatcher

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

webdupe

Rule Path
Disallow /

webemailextrac.*

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webfetcher

Rule Path
Disallow /

webfountain

Rule Path
Disallow /

webhook

Rule Path
Disallow /

webinator

Rule Path
Disallow /

weblayers

Rule Path
Disallow /

weblinker

Rule Path
Disallow /

weblog monitor

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webminer

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webpictureboss

Rule Path
Disallow /

webpictures

Rule Path
Disallow /

webquest

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webs

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webslinky

Rule Path
Disallow /

websnarf

Rule Path
Disallow /

webspider

Rule Path
Disallow /

webster

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstolperer

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

webvcr

Rule Path
Disallow /

webwalk

Rule Path
Disallow /

webwalker

Rule Path
Disallow /

webwatch

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzinger

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

wep

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget/1.5.3

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

whatuseek winona

Rule Path
Disallow /

whizg

Rule Path
Disallow /

whowhere robot

Rule Path
Disallow /

widow

Rule Path
Disallow /

wild ferret web hopper

Rule Path
Disallow /

wired digital

Rule Path
Disallow /

wisenutbot

Rule Path
Disallow /

worm

Rule Path
Disallow /

wwwc ver 0.2.5

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

w3c-checklink/3.x.x.x libwww-perl/5.xx

Rule Path
Disallow /

w3c-checklink/4.x [4.xx] libwww-perl/5.xxx

Rule Path
Disallow /

w3c_validator/1.xxx libwww-perl/5.xx

Rule Path
Disallow /

w3clinemode/5.4.0 libwww/5.x.x

Rule Path
Disallow /

w3crobot/5.4.0 libwww/5.4.0

Rule Path
Disallow /

w3m/0.x.xx

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://neteron.ru/sitemap.php
Back to top

Warnings

  • 4 invalid lines.
Back to top