rusbg.com
robots.txt

Robots Exclusion Standard data for rusbg.com

Resource Scan

Scan Details

Site Domain rusbg.com
Base Domain rusbg.com
Scan Status Ok
Last Scan2026-02-18T15:03:12+00:00
Next Scan 2026-02-25T15:03:12+00:00

Last Scan

Scanned2026-02-18T15:03:12+00:00
URL https://rusbg.com/robots.txt
Domain IPs 185.80.2.241
Response IP 185.80.2.241
Found Yes
Hash a00aa15bef6779e469d4b64c48a8131ab7579d88b710baf0e38cb9e849a38b4e
SimHash e50e50d26678

Groups

aliyunsecbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

audigentadbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-searchbot

Rule Path
Disallow /

claude-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

echoboxbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /
Allow /wirecutter/

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

jetslide

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meta-externalagent
meta-externalagent

Rule Path
Disallow /

meta-externalfetcher
meta-externalfetcher

Rule Path
Disallow /

mycentralaiscraperbot

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler
peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-user

Rule Path
Disallow /

poseidon research crawler

Rule Path
Disallow /

quillbot.com

Rule Path
Disallow /

quora-bot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

seekrbot

Rule Path
Disallow /

seznamhomepagecrawler

Rule Path
Disallow /

taragroup intelligent bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

viennatinybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

serankingbacklinksbot

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /index2.php?*
Disallow /administrator/
Disallow /cache/
Disallow /editor/
Disallow /help/
Disallow /includes/
Disallow /language/
Disallow /list/
Disallow /love/
Disallow /mambots/
Disallow /media/
Disallow /modules/
Disallow /archive/
Disallow /11/administrator/
Disallow /11/cache/
Disallow /11/component/jcomments/feed/com_content/
Disallow /11/components/
Disallow /11/includes/
Disallow /11/installation/
Disallow /11/language/
Disallow /11/libraries/
Disallow /11/modules/
Disallow /11/plugins/
Disallow /11/templates/
Disallow /11/tmp/
Disallow /11/xmlrpc/
Disallow /17/administrator/
Disallow /17/cache/
Disallow /17/component/jcomments/feed/com_content/
Disallow /17/components/
Disallow /17/includes/
Disallow /17/installation/
Disallow /17/language/
Disallow /17/libraries/
Disallow /17/modules/
Disallow /17/plugins/
Disallow /17/templates/
Disallow /17/tmp/
Disallow /17/xmlrpc/
Disallow /auto/administrator/
Disallow /auto/cache/
Disallow /auto/component/jcomments/feed/com_content/
Disallow /auto/components/
Disallow /auto/includes/
Disallow /auto/installation/
Disallow /auto/language/
Disallow /auto/libraries/
Disallow /auto/modules/
Disallow /auto/plugins/
Disallow /auto/templates/
Disallow /auto/tmp/
Disallow /auto/xmlrpc/
Disallow /bg/administrator/
Disallow /bg/cache/
Disallow /bg/component/jcomments/feed/com_content/
Disallow /bg/components/
Disallow /bg/includes/
Disallow /bg/installation/
Disallow /bg/language/
Disallow /bg/libraries/
Disallow /bg/modules/
Disallow /bg/plugins/
Disallow /bg/templates/
Disallow /bg/tmp/
Disallow /bg/xmlrpc/
Disallow /bolgaria/administrator/
Disallow /bolgaria/cache/
Disallow /bolgaria/component/jcomments/feed/com_content/
Disallow /bolgaria/components/
Disallow /bolgaria/includes/
Disallow /bolgaria/installation/
Disallow /bolgaria/language/
Disallow /bolgaria/libraries/
Disallow /bolgaria/modules/
Disallow /bolgaria/plugins/
Disallow /bolgaria/templates/
Disallow /bolgaria/tmp/
Disallow /bolgaria/xmlrpc/
Disallow /doctor/administrator/
Disallow /doctor/cache/
Disallow /doctor/component/jcomments/feed/com_content/
Disallow /doctor/components/
Disallow /doctor/includes/
Disallow /doctor/installation/
Disallow /doctor/language/
Disallow /doctor/libraries/
Disallow /doctor/modules/
Disallow /doctor/plugins/
Disallow /doctor/templates/
Disallow /doctor/tmp/
Disallow /doctor/xmlrpc/
Disallow /en/administrator/
Disallow /en/cache/
Disallow /en/component/jcomments/feed/com_content/
Disallow /en/components/
Disallow /en/includes/
Disallow /en/installation/
Disallow /en/language/
Disallow /en/libraries/
Disallow /en/modules/
Disallow /en/plugins/
Disallow /en/templates/
Disallow /en/tmp/
Disallow /en/xmlrpc/
Disallow /finance/administrator/
Disallow /finance/cache/
Disallow /finance/component/jcomments/feed/com_content/
Disallow /finance/components/
Disallow /finance/includes/
Disallow /finance/installation/
Disallow /finance/language/
Disallow /finance/libraries/
Disallow /finance/modules/
Disallow /finance/plugins/
Disallow /finance/templates/
Disallow /finance/tmp/
Disallow /finance/xmlrpc/
Disallow /internet/administrator/
Disallow /internet/cache/
Disallow /internet/component/jcomments/feed/com_content/
Disallow /internet/components/
Disallow /internet/includes/
Disallow /internet/installation/
Disallow /internet/language/
Disallow /internet/libraries/
Disallow /internet/modules/
Disallow /internet/plugins/
Disallow /internet/templates/
Disallow /internet/tmp/
Disallow /internet/xmlrpc/
Disallow /lady/administrator/
Disallow /lady/cache/
Disallow /lady/component/jcomments/feed/com_content/
Disallow /lady/components/
Disallow /lady/includes/
Disallow /lady/installation/
Disallow /lady/language/
Disallow /lady/libraries/
Disallow /lady/modules/
Disallow /lady/plugins/
Disallow /lady/templates/
Disallow /lady/tmp/
Disallow /lady/xmlrpc/
Disallow /realty/administrator/
Disallow /realty/cache/
Disallow /realty/component/jcomments/feed/com_content/
Disallow /realty/components/
Disallow /realty/includes/
Disallow /realty/installation/
Disallow /realty/language/
Disallow /realty/libraries/
Disallow /realty/modules/
Disallow /realty/plugins/
Disallow /realty/templates/
Disallow /realty/tmp/
Disallow /realty/xmlrpc/
Disallow /sport/administrator/
Disallow /sport/cache/
Disallow /sport/component/jcomments/feed/com_content/
Disallow /sport/components/
Disallow /sport/includes/
Disallow /sport/installation/
Disallow /sport/language/
Disallow /sport/libraries/
Disallow /sport/modules/
Disallow /sport/plugins/
Disallow /sport/templates/
Disallow /sport/tmp/
Disallow /sport/xmlrpc/
Disallow /travel/administrator/
Disallow /travel/cache/
Disallow /travel/component/jcomments/feed/com_content/
Disallow /travel/components/
Disallow /travel/includes/
Disallow /travel/installation/
Disallow /travel/language/
Disallow /travel/libraries/
Disallow /travel/modules/
Disallow /travel/plugins/
Disallow /travel/templates/
Disallow /travel/tmp/
Disallow /travel/xmlrpc/

Comments

  • Disallow Rules