pro.comelitgroup.com robots.txt

bingbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

5

msnbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

5

msnbot-media

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

5

yandexbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

5

abonti

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aboundex

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aboundexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

acunetix

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

afd-verbotsverfahren

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aipbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

alligator

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

allsubmitter

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

alphabot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

anarchie

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

ankit

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

apexoo

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

arquivo.pt

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

arquivo-web-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aspiegel

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

aspseek

Rule	Path
Disallow	/

Rule

Path

Disallow

/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

asterias

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

attach

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

autoemailspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

awariorssbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

awariosmartbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backdoorbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backlink-ceck

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backlink-check

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backlinkcrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backstreet

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

backweb

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

badass

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bandit

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

barkrowler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

batchftp

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

battleztar bazinga

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bbbike

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bdcbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bdfetch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

betabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bigfoot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bitacle

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

blackboard

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

black hole

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

blackwidow

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

blexbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

blow

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

blowfish

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

boardreader

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bolt

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

botalot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

brandprotect

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

brandwatch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

buddy

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

builtbottough

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

builtwith

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bullseye

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

bunnyslippers

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

buzzsumo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

calculon

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

catexplorador

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cazoodlebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ccbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cegbfeieh

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cheesebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cherrypicker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cheteam

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

chinaclaw

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

chlooe

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

claritybot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cliqzbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cloud mapping

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

coccocbot-web

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cogentbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cognitiveseo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

collector

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

com.plumanalytics

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

copier

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

copyrightcheck

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

copyscape

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cosmos

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

craftbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crawler4j

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crawler.feedback

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crawl.sogou.com

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crazywebcrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crescent

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

crunchbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

cshttp

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

curious

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

custo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

databasedrivermysqli

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

datacha0s

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dblbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

demandbase-bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

demon

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

deusu

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

devil

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

digincore

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

digitalpebble

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

diibot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dirbuster

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

disco

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

discobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

discoverybot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dispatch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dittospyder

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dnyzbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domainappender

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domaincrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domainsigmacrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domains project

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domainsproject.org

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

domainstatsbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dotbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

download wonder

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dragonfly

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

drip

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dsearch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

dts agent

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

easydl

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ebingbong

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ecatch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

eccp/1.0

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ecxi

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

eirgrabber

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

email siphon

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

email wolf

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

erocrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

evc-batch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

evil

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

exabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

express webpictures

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

extlinksbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

extractor

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

extractorpro

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

extreme picture finder

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

eyenetie

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ezooms

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

facebookscraper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

fdm

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

femtosearchbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

fhscan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

fimap

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

firefox/7.0

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

flashget

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

flunky

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

foobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

freeuploader

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

frontpage

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

fyberspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

fyrebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

galaxybot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

genieo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

germcrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

getintent

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

getright

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

getweb

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gigablast

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gigabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

g-i-g-a-b-o-t

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

go-ahead-got-it

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gotit

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gozilla

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

go!zilla

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

grabber

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

grabnet

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

grafula

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

grapefx

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

grapeshotcrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gridbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

gt::www

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

haansoft

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

haosouspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

harvest

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

havij

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

headmasterseo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

heritrix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

heritrix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

hloader

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

hmview

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

htmlparser

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

http::lite

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

httrack

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

humanlinks

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

hybridbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

iblog

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

idbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

id-search

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ilsebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

image fetch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

image sucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

indeedbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

indy library

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

infonavirobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

infotekies

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

instabid

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

intelliseek

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

interget

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

internet ninja

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

internetseer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

internetvista monitor

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ips-agent

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

iria

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

irlbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

iskanie

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

istellabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jamesbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jbrofuzz

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jennybot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jetcar

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jetty

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jikespider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

joc web spider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

joomla

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jorgee

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

justview

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

jyxobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

kenjin spider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

keyword density

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

kinza

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

kozmosbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lanshanbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

larbin

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

leechftp

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

leechget

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lexibot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lftp

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

libweb

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

libwhisker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

liebaofast

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lightspeedsystems

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

likse

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linkdexbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linkextractorpro

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linkpadbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linkscan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linksmanager

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linkwalker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linqiametadatadownloaderbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linqiarssbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

linqiascrapebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lipperhey

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lipperhey spider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

litemage_walker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lmspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lnspiderguy

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ltx71

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lwp-request

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lwp::simple

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

lwp-trivial

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

magnet

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mag-net

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

magpie-crawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mail.ru_bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

majestic12

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

majestic seo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

majestic-seo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

markmonitor

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

markwatch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

masscan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mass downloader

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mata hari

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mauibot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mb2345browser

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

meanpathbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

meanpathbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

meanpath bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mediatoolkitbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mediawords

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

megaindex.ru

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

metauri

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mfc_tear_sample

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

micromessenger

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

microsoft data access

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

microsoft url control

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

midown tool

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

miixpc

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mister pix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mj12bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mojeek

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mojolicious

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

morfeus fucking scanner

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mqqbrowser

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

mr.4x3

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

msfrontpage

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

msiecrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

msrabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

muhstik-scan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

musobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

name intelligence

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nameprotect

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

navroad

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nearsite

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

needle

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nessus

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netants

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netcraft

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netestate ne crawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netlyzer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netmechanic

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nettrack

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

net vampire

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netvibes

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

netzip

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nextgensearchbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nibbler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nicerspro

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

niki-bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nikto

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nimblecrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nimbostratus

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ninja

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nmap

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

npbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

nutch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

obot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

octopus

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

offline explorer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

offline navigator

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

oncrawl

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

openfind

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

openlinkprofiler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

openvas

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

openvas

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

oppo a33

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

orangebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

orangespider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

outclicksbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

outfoxbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pageanalyzer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

page analyzer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pagegrabber

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

page scorer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pagescorer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pandalytics

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

panscient

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

papa foto

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pavuk

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pcbrowser

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pecl::http

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

peoplepal

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

phpcrawl

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

picscout

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

picsearch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

picturefinder

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pimonster

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pi-monster

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pixray

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pleasecrawl

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

plumanalytics

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pockey

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

poe-component-client-http

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

polaris version

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

probethenet

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

propowerbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

prowebwalker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

psbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pump

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pxbroker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

pycurl

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

queryn metasearch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

quick-crawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankactive

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankactivelinkbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankflex

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankingbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankingbot2

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankivabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rankurbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

realdownload

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

reaper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rebelmouse

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

recorder

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

redesscrapy

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

reget

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

repomonkey

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

ripper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rocketcrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rogerbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

rssingbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

s1z.ru

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

salesintelligent

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sbider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

scanalert

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

scanbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

scan.lol

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

scoutjet

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

scrapy

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

screaming

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

screenerbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

searchestate

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

searchmetricsbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

semrush

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

semrushbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seokicks

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seokicks-robot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seolyticscrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seomoz

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seoprofiler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seoscanners

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seositecheckup

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

seostats

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

serpstatbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sexsearcher

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

shodan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

siphon

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sistrix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sitebeam

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

siteexplorer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

siteimprove

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sitelockspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sitesnagger

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sitesucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

site sucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sitevigil

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

slysearch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

smartdownload

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

smtbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

snake

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

snapbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

snoopy

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

socialrankiobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sociscraper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sogouspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sogou web spider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sosospider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sottopop

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spacebison

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spammen

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spankbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spanner

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sp_auditbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spinn3r

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sputnikbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

spyfu

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sqlmap

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sqlworm

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sqworm

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

steeler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

stripper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sucuri

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

superbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

superhttp

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

surfbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

surveybot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

suzuran

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

swiftbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

sysscan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

szukacz

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

t0phackteam

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

t8abot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

takeout

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

teleport

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

teleportpro

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

telesoft

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

telesphoreo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

telesphorep

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

the intraformant

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

thenomad

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

thumbor

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

tighttwatbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

titan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

toata

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

toweyabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

tracemyfile

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

trendiction

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

trendictionbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

trendiction.com

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

trendiction.de

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

true_robot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

turingos

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

turnitin

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

turnitinbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

twengabot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

twice

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

typhoeus

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

unisterbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

upflow

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

urly warning

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

urly.warning

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vacuum

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vagabondo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vb project

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vci

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vericitecrawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

vidiblescraper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

virusdie

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

voideye

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

voil

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

voltron

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wallpapers/3.0

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wallpapershd

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wasalive-bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wbsearchbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webalta

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webauto

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web auto

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webbandit

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webcollage

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web collage

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webcopier

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webdav

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webenhancer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web enhancer

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webfetch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web fetch

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webfuck

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web fuck

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webgo is

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webimagecollector

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webleacher

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webmasterworldforumbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webmeup-crawler

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webpix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web pix

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webreaper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

websauger

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web sauger

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webshag

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

websiteextractor

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

websitequester

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

website quester

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webster

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webstripper

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

websucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

web sucker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webwhacker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

webzip

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wesee

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

whack

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

whacker

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

whatweb

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

who.is bot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

widow

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

winhttrack

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wiseguys robot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wisenutbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wonderbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

woobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wotbox

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wprecon

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wpscan

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

www-collector-e

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

www-mechanize

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

www::mechanize

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

wwwoffle

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

x09mozilla

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

x22mozilla

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

xaldon webspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

xaldon_webspider

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

xenu

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

xpymep1.exe

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

youdaobot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zade

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zauba

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zauba.io

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zermelo

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zeus

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zgrab

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zitebot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zmeu

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zumbot

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

zyborg

Rule

Path

Disallow

/

Other Records

Field

Value

crawl-delay

10

Site Domain	pro.comelitgroup.com
Base Domain	comelitgroup.com
Scan Status	Ok
Last Scan	2024-11-04T13:42:10+00:00
Next Scan	2024-12-04T13:42:10+00:00

Scanned	2024-11-04T13:42:10+00:00
URL	https://pro.comelitgroup.com/robots.txt
Domain IPs	104.22.42.89, 104.22.43.89, 172.67.27.78
Response IP	104.22.43.89
Found	Yes
Hash	b7b1b457d4079af97727d8a9febe2b1acbdf6c4c9817b1604fdf000729d8586f
SimHash	634777732ba7

pro.comelitgroup.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

bingbot

Other Records

msnbot

Other Records

msnbot-media

Other Records

yandexbot

Other Records

abonti

Other Records

aboundex

Other Records

aboundexbot

Other Records

acunetix

Other Records

admantx

Other Records

afd-verbotsverfahren

Other Records

ahrefsbot

Other Records

aibot

Other Records

aihitbot

Other Records

aipbot

Other Records

alexibot

Other Records

alligator

Other Records

allsubmitter

Other Records

alphabot

Other Records

anarchie

Other Records

ankit

Other Records

apexoo

Other Records

archive.org_bot

Other Records

arquivo.pt

Other Records

arquivo-web-crawler

Other Records

aspiegel

Other Records

aspseek

Other Records

asterias

Other Records

attach

Other Records

autoemailspider

Other Records

awariorssbot

Other Records

awariosmartbot

Other Records

backdoorbot

Other Records

backlink-ceck

Other Records

backlink-check

Other Records

backlinkcrawler

Other Records

backstreet

Other Records

backweb

Other Records

badass

pro.comelitgroup.com
robots.txt