tooler.ru
robots.txt

Robots Exclusion Standard data for tooler.ru

Archived Snapshots

Resource Scan

Scan Details

Site Domain	tooler.ru
Base Domain	tooler.ru
Scan Status	Ok
Last Scan	2024-06-20T06:41:42+00:00
Next Scan	2024-07-20T06:41:42+00:00

Last Scan

Scanned	2024-06-20T06:41:42+00:00
URL	http://tooler.ru/robots.txt
Domain IPs	92.38.235.34
Response IP	92.38.235.34
Found	Yes
Hash	1039cf9f9062696582162d8006128569fb5cb2366a1de209d7c25d4f39c76dfe
SimHash	664849f04dd8

Groups

yandex

Rule	Path
Disallow	/administrator/
Disallow	/bin/
Disallow	/cli/
Disallow	/editor/
Disallow	/help/
Disallow	/includes/
Disallow	/language/
Disallow	/mambots/
Disallow	/installation/
Disallow	/stats/
Disallow	/export/
Disallow	/dump/
Disallow	/*?from=turint
Disallow	/market/
Disallow	/manuals/
Disallow	/layouts/
Disallow	/libraries/
Disallow	/logs/
Disallow	/plugins/
Disallow	/tmp/
Disallow	/*dirDesc
Disallow	/*dir%3DDesc
Disallow	tmpl%3Dcomponent
Allow	/images/
Allow	/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Rule

Path

Disallow

/administrator/

Disallow

/bin/

Disallow

/cli/

Disallow

/editor/

Disallow

/help/

Disallow

/includes/

Disallow

/language/

Disallow

/mambots/

Disallow

/installation/

Disallow

/stats/

Disallow

/export/

Disallow

/dump/

Disallow

/*?from=turint

Disallow

/market/

Disallow

/manuals/

Disallow

/layouts/

Disallow

/libraries/

Disallow

/logs/

Disallow

/plugins/

Disallow

/tmp/

Disallow

/*dirDesc

Disallow

/*dir%3DDesc

Disallow

*tmpl%3Dcomponent*

Allow

/images/

Allow

/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Other Records

Field	Value
crawl-delay	1

Field

Value

crawl-delay

googlebot

Rule	Path
Disallow	/administrator/
Disallow	/bin/
Disallow	/cli/
Disallow	/editor/
Disallow	/help/
Disallow	/includes/
Disallow	/language/
Disallow	/mambots/
Disallow	/installation/
Disallow	/stats/
Disallow	/export/
Disallow	/dump/
Disallow	/*?from=turint
Disallow	/market/
Disallow	/manuals/
Disallow	/layouts/
Disallow	/libraries/
Disallow	/logs/
Disallow	/plugins/
Disallow	/tmp/
Disallow	/*dirDesc
Disallow	/*dir%3DDesc
Disallow	tmpl%3Dcomponent
Allow	/images/
Allow	/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Rule

Path

Disallow

/administrator/

Disallow

/bin/

Disallow

/cli/

Disallow

/editor/

Disallow

/help/

Disallow

/includes/

Disallow

/language/

Disallow

/mambots/

Disallow

/installation/

Disallow

/stats/

Disallow

/export/

Disallow

/dump/

Disallow

/*?from=turint

Disallow

/market/

Disallow

/manuals/

Disallow

/layouts/

Disallow

/libraries/

Disallow

/logs/

Disallow

/plugins/

Disallow

/tmp/

Disallow

/*dirDesc

Disallow

/*dir%3DDesc

Disallow

*tmpl%3Dcomponent*

Allow

/images/

Allow

/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Other Records

Field	Value
crawl-delay	1

Field

Value

crawl-delay

*

Rule	Path
Disallow	/administrator/
Disallow	/bin/
Disallow	/cli/
Disallow	/editor/
Disallow	/help/
Disallow	/includes/
Disallow	/language/
Disallow	/mambots/
Disallow	/installation/
Disallow	/stats/
Disallow	/export/
Disallow	/dump/
Disallow	/*?from=turint
Disallow	/market/
Disallow	/manuals/
Disallow	/layouts/
Disallow	/libraries/
Disallow	/logs/
Disallow	/plugins/
Disallow	/tmp/
Allow	/images/
Disallow	/*dirDesc
Disallow	/*dir%3DDesc
Disallow	tmpl%3Dcomponent
Allow	/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Rule

Path

Disallow

/administrator/

Disallow

/bin/

Disallow

/cli/

Disallow

/editor/

Disallow

/help/

Disallow

/includes/

Disallow

/language/

Disallow

/mambots/

Disallow

/installation/

Disallow

/stats/

Disallow

/export/

Disallow

/dump/

Disallow

/*?from=turint

Disallow

/market/

Disallow

/manuals/

Disallow

/layouts/

Disallow

/libraries/

Disallow

/logs/

Disallow

/plugins/

Disallow

/tmp/

Allow

/images/

Disallow

/*dirDesc

Disallow

/*dir%3DDesc

Disallow

*tmpl%3Dcomponent*

Allow

/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Other Records

Field	Value
crawl-delay	3

Field

Value

crawl-delay

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

clickfrogbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bingbot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

idbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bingbot

Rule	Path
Disallow	/

Rule

Path

Disallow

wotbox

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-video

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-news

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-favo

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-cpro

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-ads

Rule	Path
Disallow	/

Rule

Path

Disallow

baidu

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

riddler

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

linkpadbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboardproxy

Rule	Path
Disallow	/

Rule

Path

Disallow

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

trovitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

nimbostratus-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot-sa

Rule	Path
Disallow	/

Rule

Path

Disallow

sitecheckerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

hmeonlinehomebot

Rule	Path
Disallow	/

Rule

Path

Disallow

whatcmsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

webdatastats

Rule	Path
Disallow	/

Rule

Path

Disallow

se ranking gentle bot

Rule	Path
Disallow	/

Rule

Path

Disallow

techleadzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

streton.ru analyze

Rule	Path
Disallow	/

Rule

Path

Disallow

sukibot_heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

mauibot

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou wap spider

Rule	Path
Disallow	/

Rule

Path

Disallow

tineye-bot

Rule

Path

Disallow

bubing

Rule

Path

Disallow

yeti

Rule

Path

Disallow

barkrowler

Rule

Path

Disallow

netpeakcheckerbot

Rule

Path

Disallow

dnyzbot

Rule

Path

Disallow

embedly

Rule

Path

Disallow

prlog

Rule

Path

Disallow

online-webceo-bot

Rule

Path

Disallow

twingly recon

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

struts-pwn

Rule

Path

Disallow

elefent

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

ucrawlr

Rule

Path

Disallow

proximic

Rule

Path

Disallow

researchbot

Rule

Path

Disallow

deepcrawl-ui

Rule

Path

Disallow

deepcrawl.com

Rule

Path

Disallow

spiderman

Rule

Path

Disallow

linguee bot

Rule

Path

Disallow

idg/ru

Rule

Path

Disallow

testitest1

Rule

Path

Disallow

stratagems kumo

Rule

Path

Disallow

zgrab

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

domaincrawler

Rule

Path

Disallow

embeddedwb

Rule

Path

Disallow

rukicrawler

Rule

Path

Disallow

itroot

Rule

Path

Disallow

kolahsefid.com

Rule

Path

Disallow

netseer crawler

Rule

Path

Disallow

wget

Rule

Path

Disallow

aiohttp

Rule

Path

Disallow

dataminer

Rule

Path

Disallow

bitlybot

Rule

Path

Disallow

getintent crawler

Rule

Path

Disallow

memorybot

Rule

Path

Disallow

komodiabot

Rule

Path

Disallow

anyevent-http

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

zoombot

Rule

Path

Disallow

scrapy

Rule

Path

Disallow

linkpadbot

Rule

Path

Disallow

glindahl-cocrawler

Rule

Path

Disallow

hybridbot

Rule

Path

Disallow

findcloserbot

Rule

Path

Disallow

metauri

Rule

Path

Disallow

grapeshotcrawler

Rule

Path

Disallow

begunadvertising

Rule

Path

Disallow

seochat::bot

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

dataprovider

Rule

Path

Disallow

exabot

Rule

Path

Disallow

coccoc

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

obot

Rule

Path

Disallow

sogou web spider

Rule

Path

Disallow

baiduspider

Rule

Path

Disallow

statdom.ru

Rule

Path

Disallow

guzzlehttp

Rule

Path

Disallow

arquivo-web-crawler

Rule

Path

Disallow

mixnodecache

Rule

Path

Disallow

woorankreview

Rule

Path

Disallow

tagvisit

Rule

Path

Disallow

ahrefssiteaudit

Rule

Path

Disallow

downtimedetector

Rule

Path

Disallow

nimbostratus-bot

Rule

Path

Disallow

fetchbot

Rule

Path

Disallow

jolokiapwn

Rule

Path

Disallow

megaindexauditbot

Rule

Path

Disallow

cors bot

Rule

Path

Disallow

voluumdsp-content-bot

Rule

Path

Disallow

extlinksbot

Rule

Path

Disallow

hubspot webcrawler

Rule

Path

Disallow

topvisorbot

Rule

Path

Disallow

spiderling

Rule

Path

Disallow

runet-research-crawler

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

a.pr-cy.ru

Rule

Path

Disallow

davecrawler

Rule

Path

Disallow

rookeebot

Rule

Path

Disallow

netpeakspiderbot

Rule

Path

Disallow

tiptop

Rule

Path

Disallow

grobbot

Rule

Path

Disallow

ucrawler

Rule

Path

Disallow

tobbot

Rule

Path

Disallow

ips-agent

Rule

Path

Disallow

yisouspider

Rule

Path

Disallow

leikibot

Rule

Path

Disallow

findxbot

Rule

Path

Disallow

xenu link sleuth

Rule

Path

Disallow

digincore crawler bot

Rule

Path

Disallow

gigablastopensource

Rule

Path

Disallow

funwebproducts

Rule

Path

Disallow

wecrawlforthepeace

Rule

Path

Disallow

idbot

Rule

Path

Disallow

jersey

Rule

Path

Disallow

seopultcontentanalyzer

Rule

Path

Disallow

depositfiles/filemanager

Rule

Path

Disallow

evc-batch

Rule

Path

Disallow

afb

Rule

Path

Disallow

maildbbot

Rule

Path

Disallow

laserlikebot

Rule

Path

Disallow

tsearchbot

Rule

Path

Disallow

seocharger-robot

Rule

Path

Disallow

python-requests

Rule

Path

Disallow

yoozbot

Rule

Path

Disallow

woobot

Rule

Path

Disallow

wappalyzer

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

gluten free crawler

Rule

Path

Disallow

iskanie

Rule

Path

Disallow

deusu

Rule

Path

Disallow

netcraftsurveyagent

Rule

Path

Disallow

wmtips.com

Rule

Path

Disallow

mixrankbot

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

botrobin

Rule

Path

Disallow

commoncrawler

Rule

Path

Disallow

webscore

Rule

Path

Disallow

checkmarknetwork

Rule

Path

Disallow

ravencrawler

Rule

Path

Disallow

hypestat

Rule

Path

Disallow

c-t bot

Rule

Path

Disallow

coccocbot-web

Rule

Path

Disallow

elanerubot

Rule

Path

Disallow

roboto

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

mfibot

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

tpprf

Rule

Path

Disallow

surdotlybot

Rule

Path

Disallow

toweyabot

Rule

Path

Disallow

kazbtbot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

aspiegelbot

Rule

Path

Disallow

petalbot

Rule

Path

Disallow

amazonbot

Rule

Path

Disallow

amazonbot

Product

Comment

amazonbot

Amazon's user agent

Rule

Path

Disallow

dataforseobot/1.0

Rule

Path

Disallow

dataforseobot

Rule

Path

Disallow

seekportbot

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

Other Records

Field

Value

sitemap

http://tooler.ru/index.php?option=com_xmap&view=xml&tmpl=component&id=1

sitemap

http://sadovkin.ru/index.php?option=com_xmap&view=xml&tmpl=component&id=1

sitemap

http://ulicasveta.ru/index.php?option=com_xmap&view=xml&tmpl=component&id=1

Comments

Disallow: /*/manufacturer/*
Disallow: /*/manufacturer/*
Disallow: /*/manufacturer/*

tooler.rurobots.txt

Resource Scan

Scan Details

Last Scan

Groups

yandex

Other Records

googlebot

Other Records

*

Other Records

semrushbot

clickfrogbot

mj12bot

ahrefsbot

bingbot

exabot

idbot

bingbot

wotbox

baiduspider

baiduspider-image

baiduspider-video

baiduspider-news

baiduspider-favo

baiduspider-cpro

baiduspider-ads

baidu

mj12bot

ahrefsbot

dotbot

riddler

semrushbot

linkpadbot

blexbot

flipboardproxy

aihitbot

trovitbot

velenpublicwebcrawler

dotbot

nimbostratus-bot

semrushbot-sa

sitecheckerbot

velenpublicwebcrawler

hmeonlinehomebot

whatcmsbot

webdatastats

se ranking gentle bot

techleadzbot

streton.ru analyze

sukibot_heritrix

mauibot

sogou wap spider

tineye-bot

bubing

yeti

barkrowler

netpeakcheckerbot

dnyzbot

embedly

prlog

online-webceo-bot

twingly recon

xovibot

struts-pwn

elefent

yacybot

ucrawlr

proximic

researchbot

deepcrawl-ui

deepcrawl.com

spiderman

linguee bot

idg/ru

testitest1

stratagems kumo

zgrab

ltx71

domaincrawler

tooler.ru
robots.txt