metinvest-smc.com
robots.txt

Robots Exclusion Standard data for metinvest-smc.com

Resource Scan

Scan Details

Site Domain metinvest-smc.com
Base Domain metinvest-smc.com
Scan Status Ok
Last Scan2024-09-12T19:48:22+00:00
Next Scan 2024-10-12T19:48:22+00:00

Last Scan

Scanned2024-09-12T19:48:22+00:00
URL https://metinvest-smc.com/robots.txt
Domain IPs 13.107.246.59
Response IP 13.107.246.59
Found Yes
Hash 6e2fc252f9905f7cc9c5bce6f7eb90f651691fb60e92f0e92e75dfb5f6c60862
SimHash f251314a8cf2

Groups

*

Rule Path
Disallow */search/
Disallow */admin/
Disallow */?utm_
Disallow */?product-
Disallow */checkout
Disallow */?query=
Disallow */thanks/
Disallow *%26page
Disallow *visitors
Disallow *%26amp
Disallow */ajax/
Disallow */recognition/
Disallow /html/*
Disallow */map/
Disallow */?fbclid=
Disallow */ads.txt/
Disallow */validations.json/%5C
Disallow */.well-known/
Disallow */callback/
Disallow */?gad
Disallow */payment/
Disallow */?gad
Disallow */?srsltid=

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ia_archiv

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

omgili

Rule Path
Disallow /

doc

Rule Path
Disallow /

fetch

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

hmse_robot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

larbin

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

xenu

Rule Path
Disallow /

zao

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

linko

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

obot

Rule Path
Disallow /

addthis

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

embedly

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

exabot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

bdcbot/1.0

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

twengabot-2.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

amznkassocbot/4.0

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

riddler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

psbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

hypercrawl

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

netseer

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

tdjbot

Rule Path
Disallow /

findxbot/1.0

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

y! j-asr

Rule Path
Disallow /

infoweb

Rule Path
Disallow /

nutch bots

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

unwindfetchor

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

voltron

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

silk

Rule Path
Disallow /

wget

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

linkdexbot/2.2

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

uptimebot/1.0

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

python-urllib/2.7

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

privacyawarebot/1.1

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

wget

Rule Path
Disallow /

tweetmeme

Rule Path
Disallow /

docomo/2.0

Rule Path
Disallow /

ichiro/4.0

Rule Path
Disallow /

ichiro/3.0

Rule Path
Disallow /

twengabot-discover

Rule Path
Disallow /

twengabot/2.0

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

bendercrawler

Rule Path
Disallow /

baidu

Rule Path
Disallow /

bsalsa

Rule Path
Disallow /

phpservermon/3.1.1

Rule Path
Disallow /

gluten free crawler/1.0

Rule Path
Disallow /

gluten free crawler

Rule Path
Disallow /

sogou pic spider/3.0

Rule Path
Disallow /

sogou head spider/3.0

Rule Path
Disallow /

sogou orion spider/3.0

Rule Path
Disallow /

sogou-test-spider/4.0

Rule Path
Disallow /

sogou pic agent

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

youdaobot/1.0

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://metinvest-smc.com/sitemap-ua.xml
sitemap https://metinvest-smc.com/sitemap-ru.xml

Comments

  • Dis agents

Warnings

  • 12 invalid lines.