kst.by
robots.txt

Robots Exclusion Standard data for kst.by

Archived Snapshots

Resource Scan

Scan Details

Site Domain	kst.by
Base Domain	kst.by
Scan Status	Ok
Last Scan	2024-10-25T02:42:39+00:00
Next Scan	2024-11-24T02:42:39+00:00

Last Scan

Scanned	2024-10-25T02:42:39+00:00
URL	https://kst.by/robots.txt
Domain IPs	178.172.244.13
Response IP	178.172.244.13
Found	Yes
Hash	46017b850c36fa08c160fffe8bb1e6bdf8b52b3bac5ea408800574461b820eb6
SimHash	471cdd7669a3

Groups

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seekport crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

paracrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

scrapy/1.5.0

Rule	Path
Disallow	/

Rule

Path

Disallow

scrapy

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler (velen.io)

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot/2~bl

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

pcore-http

Rule	Path
Disallow	/

Rule

Path

Disallow

the knowledge ai

Rule	Path
Disallow	/

Rule

Path

Disallow

mauibot

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler.feedback+wc@gmail.com

Rule	Path
Disallow	/

Rule

Path

Disallow

cyotekwebcopy/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

centurybot9@gmail.com

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler (crawler.feedback@gmail.com)

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler/0.7 (+http://www.exensa.com/crawl)

Rule	Path
Disallow	/

Rule

Path

Disallow

go-http-client/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

test crawl

Rule	Path
Disallow	/

Rule

Path

Disallow

scalaj-http/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

wotbox/2.01

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ebibot

Rule	Path
Disallow	/

Rule

Path

Disallow

pcore-http/v0.24.5

Rule	Path
Disallow	/

Rule

Path

Disallow

testitest1

Rule	Path
Disallow	/

Rule

Path

Disallow

vegi bot

Rule	Path
Disallow	/

Rule

Path

Disallow

istellabot/t.1

Rule	Path
Disallow	/

Rule

Path

Disallow

istellabot/t.1.13

Rule	Path
Disallow	/

Rule

Path

Disallow

istellabot

Rule	Path
Disallow	/

Rule

Path

Disallow

ltx71

Rule	Path
Disallow	/

Rule

Path

Disallow

ltx71 - (http://ltx71.com/)

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

booglebot2

Rule	Path
Disallow	/

Rule

Path

Disallow

booglebot

Rule	Path
Disallow	/

Rule

Path

Disallow

booglebot 2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

booglebot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot/v1.0.5

Rule	Path
Disallow	/

Rule

Path

Disallow

sitebot

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

influencebo

Rule	Path
Disallow	/

Rule

Path

Disallow

searchmetricsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sosospider

Rule	Path
Disallow	/

Rule

Path

Disallow

acoonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

easouspider

Rule	Path
Disallow	/

Rule

Path

Disallow

businessdbbot

Rule

Path

Disallow

superfeedr

Rule

Path

Disallow

flipboardproxy

Rule

Path

Disallow

flipboard

Rule

Path

Disallow

jikespider

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

rogerbot/1.0

Rule

Path

Disallow

flipboardproxy

Rule

Path

Disallow

swebot

Rule

Path

Disallow

swebot

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

www.80legs.com

Rule

Path

Disallow

nerdbynature.bot

Rule

Path

Disallow

comodospider

Rule

Path

Disallow

comodospider/nutch-1.2

Rule

Path

Disallow

daumoa

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

beetlebot

Rule

Path

Disallow

niki-bot

Rule

Path

Disallow

riddler

Rule

Path

Disallow

spbot

Rule

Path

Disallow

icarus6

Rule

Path

Disallow

icarus6

Rule

Path

Disallow

icarus

Rule

Path

Disallow

icarus

Rule

Path

Disallow

icarus6j

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

knelson

Rule

Path

Disallow

knelson/0.9

Rule

Path

Disallow

wotbox/2.01

Rule

Path

Disallow

blexbot/1.0

Rule

Path

Disallow

yisouspider

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

*

Rule

Path

Disallow

/*route%3Daccount/

Disallow

/*route%3Daffiliate/

Disallow

/*route%3Dcheckout/

Disallow

/*route%3Dproduct/search

Disallow

/index.php?route=product%2Fproduct*&manufacturer_id=

Disallow

/index.php?route=account%2Freturn%2Fadd

Disallow

/index.php?route=information%2Finformation&information_id=11

Disallow

/admin

Disallow

/catalog

Disallow

/system

Disallow

/*?sort=

Disallow

/*%26sort%3D

Disallow

/*?order=

Disallow

/*%26order%3D

Disallow

/*?limit=

Disallow

/*%26limit%3D

Disallow

/*?filter_name=

Disallow

/*%26filter_name%3D

Disallow

/*?filter_sub_category=

Disallow

/*%26filter_sub_category%3D

Disallow

/*?filter_description=

Disallow

/*%26filter_description%3D

Disallow

/*?tracking=

Disallow

/*%26tracking%3D

Disallow

/*compare-products

Disallow

/*search

Disallow

/cart

Disallow

/*checkout

Disallow

/*login

Disallow

/*logout

Disallow

/*vouchers

Disallow

/*wishlist

Disallow

/*my-account

Disallow

/*order-history

Disallow

/*newsletter

Disallow

/*return-add

Disallow

/*forgot-password

Disallow

/*downloads

Disallow

/*returns

Disallow

/*return

Disallow

/*returninfo

Disallow

/*loyalty

Disallow

/*transactions

Disallow

/*create-account

Disallow

/*recurring

Disallow

/*address-book

Disallow

/*reward-points

Disallow

/*affiliate-forgot-password

Disallow

/*create-affiliate-account

Disallow

/*affiliate-login

Disallow

/*affiliates

Disallow

*.html

Disallow

/*.html

Allow

/catalog/view/javascript/

Allow

/catalog/view/theme/*/

yandex

Rule

Path

Disallow

/*route%3Daffiliate/

Disallow

/*route%3Dcheckout/

Disallow

/*route%3Dproduct/search

Disallow

/index.php?route=product%2Fproduct*&manufacturer_id=

Disallow

/index.php?route=account%2Freturn%2Fadd

Disallow

/index.php?route=information%2Finformation&information_id=11

Disallow

/admin

Disallow

/catalog

Disallow

/system

Disallow

/*?sort=

Disallow

/*%26sort%3D

Disallow

/*?order=

Disallow

/*%26order%3D

Disallow

/*?limit=

Disallow

/*%26limit%3D

Disallow

/*?filter_name=

Disallow

/*%26filter_name%3D

Disallow

/*?filter_sub_category=

Disallow

/*%26filter_sub_category%3D

Disallow

/*?filter_description=

Disallow

/*%26filter_description%3D

Disallow

/*compare-products

Disallow

/*search

Disallow

/cart

Disallow

/*checkout

Disallow

/*login

Disallow

/*logout

Disallow

/*vouchers

Disallow

/*wishlist

Disallow

/*my-account

Disallow

/*order-history

Disallow

/*newsletter

Disallow

/*return-add

Disallow

/*forgot-password

Disallow

/*downloads

Disallow

/*returns

Disallow

/*return

Disallow

/*returninfo

Disallow

/*loyalty

Disallow

/*transactions

Disallow

/*create-account

Disallow

/*recurring

Disallow

/*address-book

Disallow

/*reward-points

Disallow

/*affiliate-forgot-password

Disallow

/*create-affiliate-account

Disallow

/*affiliate-login

Disallow

/*affiliates

Disallow

*.html

Disallow

/*.html

Allow

/catalog/view/javascript/

Allow

/catalog/view/theme/*/

Other Records

Field

Value

crawl-delay

0.9

googlebot

Rule

Path

Disallow

/*route%3Daffiliate/

Disallow

/*route%3Dcheckout/

Disallow

/*route%3Dproduct/search

Disallow

/index.php?route=product%2Fproduct*&manufacturer_id=

Disallow

/index.php?route=account%2Freturn%2Fadd

Disallow

/index.php?route=information%2Finformation&information_id=11

Disallow

/admin

Disallow

/catalog

Disallow

/system

Disallow

/*?sort=

Disallow

/*%26sort%3D

Disallow

/*?order=

Disallow

/*%26order%3D

Disallow

/*?limit=

Disallow

/*%26limit%3D

Disallow

/*?filter_name=

Disallow

/*%26filter_name%3D

Disallow

/*?filter_sub_category=

Disallow

/*%26filter_sub_category%3D

Disallow

/*?filter_description=

Disallow

/*%26filter_description%3D

Disallow

/*compare-products

Disallow

/*search

Disallow

/cart

Disallow

/*checkout

Disallow

/*login

Disallow

/*logout

Disallow

/*vouchers

Disallow

/*wishlist

Disallow

/*my-account

Disallow

/*order-history

Disallow

/*newsletter

Disallow

/*return-add

Disallow

/*forgot-password

Disallow

/*downloads

Disallow

/*returns

Disallow

/*return

Disallow

/*returninfo

Disallow

/*loyalty

Disallow

/*transactions

Disallow

/*create-account

Disallow

/*recurring

Disallow

/*address-book

Disallow

/*reward-points

Disallow

/*affiliate-forgot-password

Disallow

/*create-affiliate-account

Disallow

/*affiliate-login

Disallow

/*affiliates

Disallow

*.html

Disallow

/*.html

Allow

/catalog/view/javascript/

Allow

/catalog/view/theme/*/

Other Records

Field

Value

crawl-delay

0.9

mail.ru

Rule

Path

Disallow

/*route%3Daffiliate/

Disallow

/*route%3Dcheckout/

Disallow

/*route%3Dproduct/search

Disallow

/index.php?route=product%2Fproduct*&manufacturer_id=

Disallow

/index.php?route=account%2Freturn%2Fadd

Disallow

/index.php?route=information%2Finformation&information_id=11

Disallow

/admin

Disallow

/catalog

Disallow

/system

Disallow

/*?sort=

Disallow

/*%26sort%3D

Disallow

/*?order=

Disallow

/*%26order%3D

Disallow

/*?limit=

Disallow

/*%26limit%3D

Disallow

/*?filter_name=

Disallow

/*%26filter_name%3D

Disallow

/*?filter_sub_category=

Disallow

/*%26filter_sub_category%3D

Disallow

/*?filter_description=

Disallow

/*%26filter_description%3D

Disallow

/*compare-products

Disallow

/*search

Disallow

/cart

Disallow

/*checkout

Disallow

/*login

Disallow

/*logout

Disallow

/*vouchers

Disallow

/*wishlist

Disallow

/*my-account

Disallow

/*order-history

Disallow

/*newsletter

Disallow

/*return-add

Disallow

/*forgot-password

Disallow

/*downloads

Disallow

/*returns

Disallow

/*return

Disallow

/*returninfo

Disallow

/*loyalty

Disallow

/*transactions

Disallow

/*create-account

Disallow

/*recurring

Disallow

/*address-book

Disallow

/*reward-points

Disallow

/*affiliate-forgot-password

Disallow

/*create-affiliate-account

Disallow

/*affiliate-login

Disallow

/*affiliates

Disallow

*.html

Disallow

/*.html

Allow

/catalog/view/javascript/

Allow

/catalog/view/theme/*/

Other Records

Field

Value

crawl-delay

0.9

Other Records

Field

Value

sitemap

https://kst.by/sitemap_yandex.xml

sitemap

https://kst.by/sitemap.xml

sitemap

https://kst.by/sitemap.xml

Warnings

10 invalid lines.
`clean-param` is not a known field.
`host` is not a known field.

kst.byrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

dotbot

seekport crawler

paracrawl

scrapy/1.5.0

scrapy

velenpublicwebcrawler (velen.io)

velenpublicwebcrawler

semrushbot/2~bl

semrushbot

pcore-http

the knowledge ai

mauibot

crawler.feedback+wc@gmail.com

cyotekwebcopy/1.0

centurybot9@gmail.com

crawler (crawler.feedback@gmail.com)

crawler

barkrowler/0.7 (+http://www.exensa.com/crawl)

go-http-client/1.1

test crawl

scalaj-http/1.0

bubing

wotbox/2.01

ccbot/2.0

ccbot

ebibot

pcore-http/v0.24.5

testitest1

vegi bot

istellabot/t.1

istellabot/t.1.13

istellabot

ltx71

ltx71 - (http://ltx71.com/)

mj12bot

booglebot2

booglebot

booglebot 2.0

booglebot/2.0

mj12bot/v1.0.5

sitebot

baiduspider

baiduspider/2.0

influencebo

searchmetricsbot

sosospider

acoonbot

easouspider

businessdbbot

superfeedr

flipboardproxy

flipboard

jikespider

rogerbot

rogerbot/1.0

flipboardproxy

swebot

swebot

seznambot

www.80legs.com

nerdbynature.bot

comodospider

comodospider/nutch-1.2

daumoa

ahrefsbot

beetlebot

niki-bot

riddler

spbot

icarus6

icarus6

icarus

icarus

icarus6j

siteexplorer

kst.by
robots.txt