rostselmash.com
robots.txt

Robots Exclusion Standard data for rostselmash.com

Resource Scan

Scan Details

Site Domain rostselmash.com
Base Domain rostselmash.com
Scan Status Ok
Last Scan2025-05-07T07:15:27+00:00
Next Scan 2025-06-06T07:15:27+00:00

Last Scan

Scanned2025-05-07T07:15:27+00:00
URL https://rostselmash.com/robots.txt
Domain IPs 46.254.20.187
Response IP 46.254.20.187
Found Yes
Hash ce9748d5dd3c62244b0b46675da8703d8802298995de9a0b6d1221044a92bbc7
SimHash 449e1972c7b6

Groups

*

Rule Path
Disallow /economy-calc/new/
Disallow /heroes----
Disallow /de/
Disallow /en/
Disallow /fr/
Disallow /pl/
Disallow /ru/
Disallow /bitrix/
Disallow /redizine/
Allow /bitrix/components/
Allow /bitrix/cache/
Allow /bitrix/js/
Allow /bitrix/templates/
Allow /bitrix/panel/
Disallow /upload/
Allow /upload/images/
Allow /upload/resize_cache/
Disallow /search/
Disallow /feed/
Disallow /media/blog/
Disallow */subscribe/
Disallow /subscribe/
Disallow */search/
Disallow */index.php
Disallow /*show_include_exec_time%3D
Disallow /*show_page_exec_time%3D
Disallow /*show_sql_stat%3D
Disallow /*bitrix_include_areas%3D
Disallow /*clear_cache%3D
Disallow /*clear_cache_session%3D
Disallow /*?print=
Disallow /*%26print%3D
Disallow /*print_course%3D
Disallow /*?action=
Disallow /*%26action%3D
Disallow /*register%3D
Disallow /*forgot_password%3D
Disallow /*change_password%3D
Disallow /*login%3D
Disallow /*logout%3D
Disallow /*auth%3D
Disallow /*backurl%3D
Disallow /*back_url%3D
Disallow /*BACKURL%3D
Disallow /*BACK_URL%3D
Disallow /*back_url_admin%3D
Disallow /*?utm_source=
Disallow /*?sphrase_id=
Disallow /*?view_result=
Disallow /*%26view_result%3D
Disallow *?PAGEN*
Disallow *?SIZEN_1*
Disallow */?templateSection=
Disallow /?
Disallow *?
Allow *.pdf
Allow *.jpg
Allow *.JPG
Allow *.png
Allow *.css
Allow *.js

Other Records

Field Value
crawl-delay 1

ahrefssiteaudit

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

megaindex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

riddler (http://riddler.io/about)

Rule Path
Disallow /

python-urllib/2.6

Rule Path
Disallow /

disco pump

Rule Path
Disallow /

wget

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

psbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

netinfobot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

googleother

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrush

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

moz

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

serendeputy

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

seopultcontentanalyzer

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

weborama

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

keys.so

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-resty

Rule Path
Disallow /

node-fetch

Rule Path
Disallow /

okhttp

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

axios

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

fasthttp

Rule Path
Disallow /

nmap

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

riddler

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

pr-cy.ru

Rule Path
Disallow /

wp_is_mobile

Rule Path
Disallow /

alittle client

Rule Path
Disallow /

linux mozilla

Rule Path
Disallow /

paloaltonetworks

Rule Path
Disallow /

backupland

Rule Path
Disallow /

nuclei

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

kocmohabt

Rule Path
Disallow /

academicbotrtu

Rule Path
Disallow /

statdom

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://rostselmash.com/sitemap.xml