Log In Sign Up

tv.lrytas.lt
robots.txt

Robots Exclusion Standard data for tv.lrytas.lt

Resource Scan

Scan Details

Site Domain tv.lrytas.lt
Base Domain lrytas.lt
Scan Status Ok
Last Scan2025-03-10T08:03:38+00:00
Next Scan 2025-04-09T08:03:38+00:00

Last Scan

Scanned2025-03-10T08:03:38+00:00
URL https://tv.lrytas.lt/robots.txt
Redirect https://www.lrytas.lt/robots.txt
Redirect Domain www.lrytas.lt
Redirect Base lrytas.lt
Domain IPs 104.22.60.148, 104.22.61.148, 172.67.36.66
Redirect IPs 104.22.60.148, 104.22.61.148, 172.67.36.66
Response IP 104.22.61.148
Found Yes
Hash 4ba594462889125420f5ae5864be97ba68db9a8e21469c1c9e1ad6ab306ad023
SimHash 9046b312f9f8

Groups

*

Rule Path
Disallow */email.asp
Disallow */print.asp
Disallow *view%3D*

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ias-va

Rule Path
Disallow /

ias-va/3.1

Rule Path
Disallow /

ias-

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

ias-jp/3.1

Rule Path
Disallow /

ias-sg/3.1

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Rule Path
Disallow /

criteobot/0.1 (+https://www.criteo.com/criteo-crawler/)

Rule Path
Disallow /

mozilla/5.0 (compatible; dotbot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)

Rule Path
Disallow /

mozilla/5.0 (compatible;petalbot;+https://webmaster.petalsearch.com/site/petalbot)

Rule Path
Disallow /

mozilla/5.0 (compatible; linux x86_64; mail.ru_bot/2.0; +http://go.mail.ru/help/robots)

Rule Path
Disallow /

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

Rule Path
Disallow /

mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)

Rule Path
Disallow /

cutbot; 1.5; http://cutbot.net/

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/7.0; +http://ahrefs.com/robot/)

Rule Path
Disallow /

mozilla/5.0 (compatible; blexbot/1.0; +http://webmeup-crawler.com/)

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /
Back to top

Comments

  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block trendkite-akashic-crawler
  • Block ahrefs bot
Back to top