it-s.lv robots.txt

*

Rule	Path	Comment
Disallow	/add-to-cart%3D	-
Disallow	/cart/	-
Disallow	/checkout/	-
Disallow	/my-account/	-
Disallow	/wp-admin/	block access to admin section
Disallow	/wp-login.php	block access to admin section
Disallow	/search/	block access to internal search result pages
Disallow	?s=	block access to internal search result pages
Disallow	?p=	block access to pages for which permalinks fails
Disallow	%26p%3D	block access to pages for which permalinks fails
Disallow	%26preview%3D	block access to preview pages
Disallow	/tag/	block access to tag pages
Disallow	/author/	block access to author pages
Disallow	/404-error/	block access to 404 page
Disallow	/admin	-

Rule

Path

Comment

Disallow

/*add-to-cart%3D*

-

Disallow

/cart/

-

Disallow

/checkout/

-

Disallow

/my-account/

-

Disallow

/wp-admin/

block access to admin section

Disallow

/wp-login.php

block access to admin section

Disallow

/search/

block access to internal search result pages

Disallow

*?s=*

block access to internal search result pages

Disallow

*?p=*

block access to pages for which permalinks fails

Disallow

*%26p%3D*

block access to pages for which permalinks fails

Disallow

*%26preview%3D*

block access to preview pages

Disallow

/tag/

block access to tag pages

Disallow

/author/

block access to author pages

Disallow

/404-error/

block access to 404 page

Disallow

/admin

-

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

facebookbot

Product	Comment
facebookbot	rules targeted at FacebookBot

Product

Comment

facebookbot

rules targeted at FacebookBot

Rule	Path	Comment
Disallow	/nocrawl	disallow this directory

Rule

Path

Comment

Disallow

/nocrawl

disallow this directory

aacrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

/

asterias

Rule	Path
Disallow	/

Rule

Path

Disallow

/

atomic_email_hunter

Rule	Path
Disallow	/

Rule

Path

Disallow

/

b2w/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider+

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-cpro

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-favo

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-image

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-mobile

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-news

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-sfkr

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider-video

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bookmark search tool

Rule	Path
Disallow	/

Rule

Path

Disallow

/

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

/

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

/

check url bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cheesebot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cherrypicker

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cityreview

Rule	Path
Disallow	/

Rule

Path

Disallow

/

copernic

Rule	Path
Disallow	/

Rule

Path

Disallow

/

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cosmos

Rule	Path
Disallow	/

Rule

Path

Disallow

/

crescent

Rule	Path
Disallow	/

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

cyberpatrol sitecat webbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

dittospyder

Rule	Path
Disallow	/

Rule

Path

Disallow

/

domainwatcher

Rule	Path
Disallow	/

Rule

Path

Disallow

/

emailcollector

Rule	Path
Disallow	/

Rule

Path

Disallow

/

emailsiphon

Rule	Path
Disallow	/

Rule

Path

Disallow

/

emailwolf

Rule	Path
Disallow	/

Rule

Path

Disallow

/

erocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

exooba

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

ezooms/1.0

Rule

Path

Disallow

/

fairad client

Rule

Path

Disallow

/

fasterfox

Rule

Path

Disallow

/

flaming attackbot

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

gaisbot

Rule

Path

Disallow

/

getright/4.2

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

grapeshotcrawler/2.0

Rule

Path

Disallow

/

grapefx/0.9

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

httrack 3.0

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

ia_archiver/1.6

Rule

Path

Disallow

/

iccrawler - icjobs

Rule

Path

Disallow

/

infohelfer/1.3.0

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

iron33/1.0.2

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

jobs.de-robot

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

keyword density/0.9

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

mail.ru

Rule

Path

Disallow

/

mail.ru_bot/2.0

Rule

Path

Disallow

/

magpie-crawler/1.1

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

mj12bot/v1.4.3

Rule

Path

Disallow

/

mlbot

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

onetszukaj

Rule

Path

Disallow

/

openbot

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openfind data gatherer

Rule

Path

Disallow

/

oracle ultra search

Rule

Path

Disallow

/

perman

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

psbot

Rule

Path

Disallow

/

python-urllib

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

radiation retriever 1.1

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

scoutjet

Rule

Path

Disallow

/

seznambot/3.0

Rule

Path

Disallow

/

searchpreview

Rule

Path

Disallow

/

spbot/3.1

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

sosospider/2.0

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

toplistbot

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

true_robot/1.0

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

turnitinbot/1.5

Rule

Path

Disallow

/

twiceler

Rule

Path

Disallow

/

twiceler.

Rule

Path

Disallow

/

url control

Rule

Path

Disallow

/

urlspion

Rule

Path

Disallow

/

url_spider_pro

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

webalta crawler

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webbericht.com ?

Rule

Path

Disallow

/

webcapture 2.0

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webcopier v.2.2

Rule

Path

Disallow

/

webcopier v3.2a

Rule

Path

Disallow

/

webdatacentrebot

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webwiki

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

webzip/4.21

Rule

Path

Disallow

/

webzip/5.0

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

xenuвґs

Rule

Path

Disallow

/

xenuвґs link sleuth 1.1c

Rule

Path

Disallow

/

yandex

Rule

Path

Disallow

/

yandexbot/3.0

Rule

Path

Disallow

/

yeti

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus link scout

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semanticscholarbot

Rule

Path

Disallow

/

petalbot

Rule

Path

Disallow

/

Site Domain	it-s.lv
Base Domain	it-s.lv
Scan Status	Ok
Last Scan	2024-11-14T07:19:34+00:00
Next Scan	2024-11-21T07:19:34+00:00

Scanned	2024-11-14T07:19:34+00:00
URL	https://www.it-s.lv/robots.txt
Domain IPs	62.63.134.2
Response IP	62.63.134.2
Found	Yes
Hash	d87abdf216f37ef5539e6399ab73633b20e4e716feb9e8403d8a09bfdc5d6fb6
SimHash	77355282c932

it-s.lvrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

amazonbot

facebookbot

aacrawler

ahrefsbot

alexibot

aqua_products

asterias

atomic_email_hunter

b2w/0.1

backdoorbot/1.0

backlinkcrawler

baiduspider

baiduspider+

baiduspider-cpro

baiduspider-favo

baiduspider-image

baiduspider-mobile

baiduspider-news

baiduspider-sfkr

baiduspider-video

blowfish/1.0

bookmark search tool

botalot

botrighthere

builtbottough

bullseye/1.0

bunnyslippers

check url bot

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

cityreview

copernic

copyrightcheck

cosmos

crescent

crescent internet toolpak http ole control v.1.0

cyberpatrol sitecat webbot

dittospyder

domainwatcher

emailcollector

emailsiphon

emailwolf

erocrawler

exabot

exooba

extractorpro

ezooms/1.0

fairad client

fasterfox

flaming attackbot

foobot

gaisbot

getright/4.2

gigabot

grapeshotcrawler/2.0

grapefx/0.9

harvest/1.5

heritrix

hloader

httplib

httrack

httrack 3.0

humanlinks

ia_archiver

ia_archiver/1.6

iccrawler - icjobs

infohelfer/1.3.0

infonavirobot

iron33/1.0.2

jennybot

jobs.de-robot

kenjin spider

it-s.lv
robots.txt