Log In Sign Up

livejournal.ru
robots.txt

Robots Exclusion Standard data for livejournal.ru

Resource Scan

Scan Details

Site Domain livejournal.ru
Base Domain livejournal.ru
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-29T00:09:25+00:00
Next Scan 2024-11-28T00:09:25+00:00

Last Successful Scan

Scanned2024-08-01T00:06:30+00:00
URL https://livejournal.ru/robots.txt
Redirect https://www.livejournal.com/robots.txt
Redirect Domain www.livejournal.com
Redirect Base livejournal.com
Domain IPs 81.19.74.40, 81.19.74.41
Redirect IPs 81.19.74.0, 81.19.74.1
Response IP 81.19.74.1
Found Yes
Hash 56c6734a2f3f5cc47831b8561a13a6a1bd1cbc16aea19a0ecaf4993f922b3f01
SimHash 691d8c6205ed

Groups

yandex

Rule Path
Allow /
Disallow /allpics.bml
Disallow /update.bml
Disallow /identity
Disallow /login.bml
Disallow /manage
Disallow /poll
Disallow /profile
Disallow /schools
Disallow /todo
Disallow /tools
Disallow /update.bml
Disallow /userinfo.bml
Disallow /users
Allow /ratings/$
Disallow /ratings
Disallow /syn
Disallow /latest
Disallow /ljtimes
Disallow /talkread
Disallow /inbox
Disallow /misc
Disallow /legal
Disallow /checklistposts
Disallow /away
Disallow /rsearch
Disallow /gsearch

spbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /allpics.bml
Disallow /update.bml
Disallow /identity
Disallow /login.bml
Disallow /manage
Disallow /poll
Disallow /profile
Disallow /schools
Disallow /todo
Disallow /tools
Disallow /update.bml
Disallow /userinfo.bml
Disallow /users
Allow /ratings/$
Disallow /ratings
Disallow /syn
Disallow /latest
Disallow /ljtimes
Disallow /talkread
Disallow /inbox
Disallow /misc
Disallow /legal
Disallow /checklistposts
Disallow /away
Disallow /rsearch
Disallow /gsearch
Back to top

Other Records

Field Value
sitemap https://www.livejournal.com/sitemap.xml
Back to top

Comments

  • Blocked journals aren't listed here because robots.txt files
  • can't be above 50k or so, depending on the spider.
  • Instead, blocked journals have HTML inserted in them which
  • should prevent behaved spiders from indexing it.
  • Note that http://username.livejournal.com journals have an
  • autogenerated robots.txt, since it can be small.
Back to top

Warnings

  • `clean-param` is not a known field.
  • `host` is not a known field.
Back to top