mnews.net.ua
robots.txt

Robots Exclusion Standard data for mnews.net.ua

Archived Snapshots

Resource Scan

Scan Details

Site Domain	mnews.net.ua
Base Domain	mnews.net.ua
Scan Status	Failed
Failure Reason	Scan timed out.
Last Scan	2024-09-12T11:59:21+00:00
Next Scan	2024-12-11T11:59:21+00:00

Last Successful Scan

Scanned	2024-04-23T00:12:04+00:00
URL	https://www.mnews.net.ua/robots.txt
Domain IPs	104.21.59.186, 172.67.182.112, 2606:4700:3032::ac43:b670, 2606:4700:3037::6815:3bba
Response IP	172.67.182.112
Found	Yes
Hash	d1e6eb406de26af49bedc6d7109c147147132f9aec0a8c1cc74cc1aa482b836f
SimHash	7134234beeb2

Groups

*

Rule	Path
Disallow	/wp-admin/
Disallow	/*?s=
Disallow	/search/
Disallow	/?
Disallow	utm%3D
Disallow	*openstat%3D

Rule

Path

Disallow

/wp-admin/

Disallow

/*?s=

Disallow

/search/

Disallow

*utm*%3D

Disallow

*openstat%3D

addthis

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexabot

Rule	Path
Disallow	/

Rule

Path

Disallow

amznkassocbot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

baidu

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

bdcbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bdcbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bsalsa

Rule	Path
Disallow	/

Rule

Path

Disallow

butterfly

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

changedetection

Rule	Path
Disallow	/

Rule

Path

Disallow

coccoc

Rule	Path
Disallow	/

Rule

Path

Disallow

crazywebcrawler-spider

Rule	Path
Disallow	/

Rule

Path

Disallow

daumoa

Rule	Path
Disallow	/

Rule

Path

Disallow

docomo/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

easouspider

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms

Rule	Path
Disallow	/

Rule

Path

Disallow

findxbot

Rule	Path
Disallow	/

Rule

Path

Disallow

findxbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboard

Rule	Path
Disallow	/

Rule

Path

Disallow

genieo

Rule	Path
Disallow	/

Rule

Path

Disallow

gluten free crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

haosouspider

Rule	Path
Disallow	/

Rule

Path

Disallow

hypercrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

ichiro

Rule	Path
Disallow	/

Rule

Path

Disallow

ichiro/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ichiro/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

infoweb

Rule	Path
Disallow	/

Rule

Path

Disallow

istellabot

Rule	Path
Disallow	/

Rule

Path

Disallow

libwww-perl

Rule	Path
Disallow	/

Rule

Path

Disallow

linguee bot

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot/2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot/2.2

Rule	Path
Disallow	/

Rule

Path

Disallow

linkpadbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ltx71

Rule	Path
Disallow	/

Rule

Path

Disallow

meanpathbot

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

naverbot

Rule	Path
Disallow	/

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

netseer

Rule

Path

Disallow

nutch bots

Rule

Path

Disallow

obot

Rule

Path

Disallow

paperlibot

Rule

Path

Disallow

phpservermon/3.1.1

Rule

Path

Disallow

privacyawarebot/1.1

Rule

Path

Disallow

proximic

Rule

Path

Disallow

psbot

Rule

Path

Disallow

queryseekerspider

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

screenerbot

Rule

Path

Disallow

searchmetricsbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

sf/2 (skipfish)

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

sogou head spider/3.0

Rule

Path

Disallow

sogou orion spider/3.0

Rule

Path

Disallow

sogou pic agent

Rule

Path

Disallow

sogou pic spider/3.0

Rule

Path

Disallow

sputnikbot

Rule

Path

Disallow

sunrise

Rule

Path

Disallow

surdotlybot

Rule

Path

Disallow

surdotlybot

Rule

Path

Disallow

swiftbot

Rule

Path

Disallow

tdjbot

Rule

Path

Disallow

trovitbot

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

tweetmeme

Rule

Path

Disallow

tweetmemebot

Rule

Path

Disallow

twengabot

Rule

Path

Disallow

twengabot-2.0

Rule

Path

Disallow

twengabot-discover

Rule

Path

Disallow

twengabot/2.0

Rule

Path

Disallow

unwindfetchor

Rule

Path

Disallow

uptimebot

Rule

Path

Disallow

uptimebot/1.0

Rule

Path

Disallow

uptimerobot

Rule

Path

Disallow

uptimerobot/2.0

Rule

Path

Disallow

voltron

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

wesee

Rule

Path

Disallow

wget

Rule

Path

Disallow

wget

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

y! j-asr

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

yeti

Rule

Path

Disallow

yodaobot

Rule

Path

Disallow

yoozbot-2.2

Rule

Path

Disallow

youdaobot/1.0

Rule

Path

Disallow

yahoo! slurp

Rule

Path

Disallow

yahoo-slurp

Rule

Path

Disallow

anews/1.0

Rule

Path

Disallow

anews

Rule

Path

Disallow

kribrum

Rule

Path

Disallow

kribrum/0.1

Rule

Path

Disallow

iskanie

Rule

Path

Disallow

paperlibot/2.1

Rule

Path

Disallow

special_archiver/3.1.1

Rule

Path

Disallow

special_archiver

Rule

Path

Disallow

ccbot/2.0

Rule

Path

Disallow

exabot/3.0

Rule

Path

Disallow

semrushbot/1.1~bl

Rule

Path

Disallow

semrushbot/1.1

Rule

Path

Disallow

semrushbot-sa/0.97

Rule

Path

Disallow

um-fc

Rule

Path

Disallow

um-fc/1.0

Rule

Path

Disallow

um-ln

Rule

Path

Disallow

um-ln/1.0

Rule

Path

Disallow

heritrix

Rule

Path

Disallow

deusu/5.0.2

Rule

Path

Disallow

bitlybot

Rule

Path

Disallow

bitlybot/3.0

Rule

Path

Disallow

maxpointcrawler

Rule

Path

Disallow

getintent crawler

Rule

Path

Disallow

scrapy

Rule

Path

Disallow

scrapy/1.2.2

Rule

Path

Disallow

twitterbot

Rule

Path

Disallow

twitterbot/1.0

Rule

Path

Disallow

sputnikbot/2.3

Rule

Path

Disallow

trident/6.0

Rule

Path

Disallow

trident/5.0

Rule

Path

Disallow

trident

Rule

Path

Disallow

guzzlehttp/6.1.0

Rule

Path

Disallow

guzzlehttp

Rule

Path

Disallow

x11

Rule

Path

Disallow

msnbot

Rule

Path

Disallow

bingbot

Rule

Path

Disallow

bingbot/2.0

Rule

Path

Disallow

stackrambler

Rule

Path

Disallow

metaspider

Rule

Path

Disallow

aport

Rule

Path

Disallow

scooter

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

trendictionbot0.5.0

Rule

Path

Disallow

kraken

Rule

Path

Disallow

kraken/0.1

Rule

Path

Disallow

dotbot/1.1

Rule

Path

Disallow

bubing

Rule

Path

Disallow

yeti/1.1

Rule

Path

Disallow

seznambot/3.2

Rule

Path

Disallow

webartexbot

Rule

Path

Disallow

blexbot/1.0

Rule

Path

Disallow

spbot/5.0.3

Rule

Path

Disallow

spbot

Rule

Path

Disallow

um-ic

Rule

Path

Disallow

um-ic/1.0

Rule

Path

Disallow

germcrawler

Rule

Path

Disallow

sogou

Rule

Path

Disallow

gingercrawler

Rule

Path

Disallow

nutch

Rule

Path

Disallow

urlbot

Rule

Path

Disallow

scoutjet

Rule

Path

Disallow

grouphigh

Rule

Path

Disallow

grouphigh/1.0

Rule

Path

Disallow

mj12bot/v1.4.3

Rule

Path

Disallow

mj12bot/v1.4.0

Rule

Path

Disallow

mj12bot/v1.2.4

Rule

Path

Disallow

mj12bot/v1.2.3

Rule

Path

Disallow

mj12bot/v1.0.8

Rule

Path

Disallow

mj12bot/v1.0.7

Rule

Path

Disallow

mj12bot/v1.0.6

Rule

Path

Disallow

mj12bot/v1.0.5

Rule

Path

Disallow

mj12bot/v1.4.5

Rule

Path

Disallow

mj12bot/v1.4.6

Rule

Path

Disallow

mj12bot/v1.4.7

Rule

Path

Disallow

mj12bot/v1.4.8

Rule

Path

Disallow

rogerbot/1.0

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

megaindex

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

megaindex

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

steeler/3.5

Rule

Path

Disallow

steeler

Rule

Path

Disallow

ahrefsbot/5.1

Rule

Path

Disallow

ahrefsbot/5.2

Rule

Path

Disallow

ahrefsbot/1.0

Rule

Path

Disallow

ahrefsbot/2.0

Rule

Path

Disallow

ahrefsbot/3.0

Rule

Path

Disallow

ahrefsbot/4.0

Rule

Path

Disallow

ahrefsbot/5.0

Rule

Path

Disallow

ahrefsbot/3.1

Rule

Path

Disallow

serpstatbot

Rule

Path

Disallow

boitho

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://mnews.net.ua/sitemap_index.xml

Warnings

12 invalid lines.

mnews.net.uarobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

addthis

ahrefsbot

alexabot

amznkassocbot/4.0

baidu

baiduspider

baiduspider-image

bdcbot

bdcbot/1.0

blexbot

bsalsa

butterfly

ccbot

changedetection

coccoc

crazywebcrawler-spider

daumoa

docomo/2.0

dotbot

easouspider

exabot

ezooms

findxbot

findxbot/1.0

flipboard

genieo

gluten free crawler

grapeshot

haosouspider

hypercrawl

ichiro

ichiro/3.0

ichiro/4.0

infoweb

istellabot

libwww-perl

linguee bot

linkdexbot

linkdexbot/2.0

linkdexbot/2.1

linkdexbot/2.2

linkpadbot

ltx71

meanpathbot

megaindex.ru

megaindex.ru/2.0

naverbot

nerdybot

netseer

nutch bots

obot

paperlibot

phpservermon/3.1.1

privacyawarebot/1.1

proximic

psbot

queryseekerspider

qwantify

rogerbot

screenerbot

searchmetricsbot

semrushbot-sa

sf/2 (skipfish)

siteexplorer

sogou head spider/3.0

sogou orion spider/3.0

sogou pic agent

sogou pic spider/3.0

sputnikbot

sunrise

surdotlybot

surdotlybot

swiftbot

tdjbot

mnews.net.ua
robots.txt