webformula.pro
robots.txt

Robots Exclusion Standard data for webformula.pro

Resource Scan

Scan Details

Site Domain webformula.pro
Base Domain webformula.pro
Scan Status Ok
Last Scan2025-11-15T09:39:52+00:00
Next Scan 2025-12-15T09:39:52+00:00

Last Scan

Scanned2025-11-15T09:39:52+00:00
URL https://webformula.pro/robots.txt
Redirect https://www.webformula.pro/robots.txt
Redirect Domain www.webformula.pro
Redirect Base webformula.pro
Domain IPs 5.183.189.45
Redirect IPs 5.183.189.45
Response IP 5.183.189.45
Found Yes
Hash 05cb8cdecd922782bbc0e86882768169861bd49a20914a85e3734b9c1102a759
SimHash 44561970e786

Groups

*

Rule Path
Disallow */index.php
Disallow /bitrix/
Disallow /*show_include_exec_time%3D
Disallow /*show_page_exec_time%3D
Disallow /*show_sql_stat%3D
Disallow /*bitrix_include_areas%3D
Disallow /*clear_cache%3D
Disallow /*clear_cache_session%3D
Disallow /*ADD_TO_COMPARE_LIST
Disallow /*ORDER_BY
Disallow /*PAGEN
Disallow /*?print=
Disallow /*%26print%3D
Disallow /*print_course%3D
Disallow /*?action=
Disallow /*%26action%3D
Disallow /*register%3D
Disallow /*forgot_password%3D
Disallow /*change_password%3D
Disallow /*login%3D
Disallow /*logout%3D
Disallow /*auth%3D
Disallow /*backurl%3D
Disallow /*back_url%3D
Disallow /*BACKURL%3D
Disallow /*BACK_URL%3D
Disallow /*back_url_admin%3D
Disallow /*?utm_source=
Disallow /*?bxajaxid=
Disallow /*%26bxajaxid%3D
Disallow /*?view_result=
Disallow /*%26view_result%3D
Allow /bitrix/components/
Allow /bitrix/cache/
Allow /bitrix/js/
Allow /bitrix/templates/
Allow /bitrix/panel/

ahrefssiteaudit

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

megaindex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

riddler (http://riddler.io/about)

Rule Path
Disallow /

python-urllib/2.6

Rule Path
Disallow /

disco pump

Rule Path
Disallow /

wget

Rule Path
Disallow /

teleport pro

Rule Path
Disallow /

psbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

netinfobot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

googleother

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrush

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

moz

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

serendeputy

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

seopultcontentanalyzer

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

weborama

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

keys.so

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-resty

Rule Path
Disallow /

node-fetch

Rule Path
Disallow /

okhttp

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

axios

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

fasthttp

Rule Path
Disallow /

nmap

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

riddler

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

pr-cy.ru

Rule Path
Disallow /

wp_is_mobile

Rule Path
Disallow /

alittle client

Rule Path
Disallow /

linux mozilla

Rule Path
Disallow /

paloaltonetworks

Rule Path
Disallow /

backupland

Rule Path
Disallow /

nuclei

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

kocmohabt

Rule Path
Disallow /

academicbotrtu

Rule Path
Disallow /

statdom

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.webformula.pro/sitemap.xml

Warnings

  • `host` is not a known field.