Log In Sign Up

clever-logistics.ru
robots.txt

Robots Exclusion Standard data for clever-logistics.ru

Resource Scan

Scan Details

Site Domain clever-logistics.ru
Base Domain clever-logistics.ru
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-29T06:11:39+00:00
Next Scan 2024-10-27T06:11:39+00:00

Last Successful Scan

Scanned2023-06-13T04:51:45+00:00
URL https://clever-logistics.ru/robots.txt
Domain IPs 104.22.8.154, 104.22.9.154, 172.67.23.247, 2606:4700:10::6816:89a, 2606:4700:10::6816:99a, 2606:4700:10::ac43:17f7
Response IP 104.22.9.154
Found Yes
Hash d21e7f761e9bcb67be966d54d3e2c706eaccc3e3e8ef8592bef7cd8eb3c2564b
SimHash f8deb142aa79

Groups

yandex

Rule Path
Disallow /*?*
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.xls$
Disallow /*.xlsx$
Disallow /*.ppt$
Disallow /*?utm*
Disallow /*?openstat*
Disallow /*?gclid*
Disallow /net*
Disallow /allottedLanding?*
Disallow /ext/kontejnernye-perevozki
Disallow /signin

Other Records

Field Value
crawl-delay 2.0

googlebot

Rule Path
Disallow /*?*
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.xls$
Disallow /*.xlsx$
Disallow /*.ppt$
Disallow /*?utm*
Disallow /*?openstat*
Disallow /*?gclid*
Disallow /net*
Disallow /allottedLanding?*
Disallow /ext/kontejnernye-perevozki
Disallow /signin

Other Records

Field Value
crawl-delay 2.0

*

Rule Path
Disallow /*?*
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.xls$
Disallow /*.xlsx$
Disallow /*.ppt$
Disallow /*?utm*
Disallow /*?openstat*
Disallow /*?gclid*
Disallow /net*
Disallow /allottedLanding?*
Disallow /ext/kontejnernye-perevozki
Disallow /signin

Other Records

Field Value
crawl-delay 2.0

addthis

Rule Path
Disallow /

baidu

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

exabot

Rule Path
Disallow /

findxbot/1.0

Rule Path
Disallow /

genieo

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ichiro/4.0

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

larbin

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

mfe

Rule Path
Disallow /

moget

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

ning

Rule Path
Disallow /

obot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

privacyawarebot/1.1

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

scribdreader

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

sogou orion spider/3.0

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou-test-spider/4.0

Rule Path
Disallow /

sunrise

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

twengabot-discover

Rule Path
Disallow /

uptimebot/1.0

Rule Path
Disallow /

voltron

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

y! j-asr

Rule Path
Disallow /

wget

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://clever-logistics.ru/sitemap.xml
Back to top

Comments

  • Clean-param:
  • User-agent: Alexabot
  • Disallow: /
  • User-agent: MegaIndex.ru
  • Disallow: /
Back to top

Warnings

  • 4 invalid lines.
Back to top