Log In Sign Up

dotcom.co.il
robots.txt

Robots Exclusion Standard data for dotcom.co.il

Resource Scan

Scan Details

Site Domain dotcom.co.il
Base Domain dotcom.co.il
Scan Status Ok
Last Scan2024-11-13T16:49:14+00:00
Next Scan 2024-11-20T16:49:14+00:00

Last Scan

Scanned2024-11-13T16:49:14+00:00
URL https://dotcom.co.il/robots.txt
Redirect https://www.dotcom.co.il/robots.txt
Redirect Domain www.dotcom.co.il
Redirect Base dotcom.co.il
Domain IPs 173.236.207.0
Redirect IPs 173.236.207.0
Response IP 173.236.207.0
Found Yes
Hash fb64d199449f5c9f9ae1a0b09a1e740beb50a0ede238b4009571051ec9159f29
SimHash 197256eaa6cf

Groups

a1 sitemap generator

Rule Path
Disallow /

a1 website download

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

aasp

Rule Path
Disallow /

abachobot

Rule Path
Disallow /

achulkov.net page walker

Rule Path
Disallow /

aesop_com_spiderman

Rule Path
Disallow /

aespider

Rule Path
Disallow /

af knowledge now verity spider

Rule Path
Disallow /

aibot

Rule Path
Disallow /

aisiid

Rule Path
Disallow /

aitcsrobot/1.1

Rule Path
Disallow /

amznkassocbot

Rule Path
Disallow /

aonde-spider

Rule Path
Disallow /

asaha search engine turkey

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

aspider/0.09

Rule Path
Disallow /

auresys

Rule Path
Disallow /

avsearch-

Rule Path
Disallow /

abonti

Rule Path
Disallow /

abotemailsearc

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

accmonitor compliance server

Rule Path
Disallow /

acme.spider

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

activetouristbot

Rule Path
Disallow /

ad muncher

Rule Path
Disallow /

admedia bot

Rule Path
Disallow /

adamm bot,

Rule Path
Disallow /

advanced email extractor

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot/5.0

Rule Path
Disallow /

akamai-sitesnapshot

Rule Path
Disallow /

alexawebsearchplatform

Rule Path
Disallow /

alexfdownload

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alkalinebot

Rule Path
Disallow /

all acronyms bot

Rule Path
Disallow /

alpha search agent

Rule Path
Disallow /

ampmppc.com

Rule Path
Disallow /

amerla search bot

Rule Path
Disallow /

amfibibot

Rule Path
Disallow /

anemone

Rule Path
Disallow /

anonymous/3g bot

Rule Path
Disallow /

anonymouse.org

Rule Path
Disallow /

anotherbot

Rule Path
Disallow /

answerbot

Rule Path
Disallow /

answerbus

Rule Path
Disallow /

answerchase prove

Rule Path
Disallow /

antbot

Rule Path
Disallow /

antisantyworm

Rule Path
Disallow /

antro.net

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

apache-httpclient/4.0.1 (java 1.5)

Rule Path
Disallow /

apachebench

Rule Path
Disallow /

aport

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

arabot

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

arikus_spider

Rule Path
Disallow /

art-online.com

Rule Path
Disallow /

artavisbot

Rule Path
Disallow /

artera

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

astrofind

Rule Path
Disallow /

atlocalbot

Rule Path
Disallow /

atomic_email_hunter

Rule Path
Disallow /

attributor.combot

Rule Path
Disallow /

attributor/dejan-

Rule Path
Disallow /

autobaron crawler

Rule Path
Disallow /

axonize-bot

Rule Path
Disallow /

ayna

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

biglotron

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blt

Rule Path
Disallow /

boia-scan-agent

Rule Path
Disallow /

boia.org-scan-agent

Rule Path
Disallow /

boi_crawl_00

Rule Path
Disallow /

bot

Rule Path
Disallow /

braintime_search

Rule Path
Disallow /

bsearchr&d

Rule Path
Disallow /

bspider

Rule Path
Disallow /

byindia

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backrub

Rule Path
Disallow /

backstreet browser

Rule Path
Disallow /

backweb

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

bandit

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

batchftp

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

becomejpbot

Rule Path
Disallow /

beetlebot

Rule Path
Disallow /

bender

Rule Path
Disallow /

big brother

Rule Path
Disallow /

big data

Rule Path
Disallow /

bigcliquebot

Rule Path
Disallow /

bigcliquebot

Rule Path
Disallow /

bigado.com

Rule Path
Disallow /

bigfoot

Rule Path
Disallow /

bilbo

Rule Path
Disallow /

bilgibetabot

Rule Path
Disallow /

bilgibot

Rule Path
Disallow /

bitvouseragent

Rule Path
Disallow /

bizbot04 kirk.overleaf.com

Rule Path
Disallow /

bizworks retriever

Rule Path
Disallow /

bizbot003

Rule Path
Disallow /

black hole

Rule Path
Disallow /

black.hole

Rule Path
Disallow /

blackmask.net search engine

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blackbird

Rule Path
Disallow /

blaiz-bee

Rule Path
Disallow /

blinkx/dfs-fetch

Rule Path
Disallow /

blitzbot

Rule Path
Disallow /

blog conversation project

Rule Path
Disallow /

blogmyway

Rule Path
Disallow /

blogpulselive

Rule Path
Disallow /

blogrefsbot

Rule Path
Disallow /

blogscope

Rule Path
Disallow /

blogslive

Rule Path
Disallow /

bloobybot

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

boaconstrictor

Rule Path
Disallow /

boardreader-image-fetcher

Rule Path
Disallow /

bookmark buddy bookmark checker

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

bot

Rule Path
Disallow /

bot apoena

Rule Path
Disallow /

botalot

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

botswana

Rule Path
Disallow /

bpbot

Rule Path
Disallow /

brokenlinkcheck.com

Rule Path
Disallow /

browseremulator

Rule Path
Disallow /

browsermob

Rule Path
Disallow /

bruinbot

Rule Path
Disallow /

btsearch

Rule Path
Disallow /

buddy

Rule Path
Disallow /

buibui

Rule Path
Disallow /

buildcms crawler

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

buscadorclarin

Rule Path
Disallow /

buscaplus robi

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

buyhawaiibot

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

byspider

Rule Path
Disallow /

bytedance

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ccgcrawl

Rule Path
Disallow /

cd-preload

Rule Path
Disallow /

ce-preload

Rule Path
Disallow /

cert figleafbot

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

coast webmaster pro

Rule Path
Disallow /

coast scan engine

Rule Path
Disallow /

cacheblaster

Rule Path
Disallow /

caddbot

Rule Path
Disallow /

cafi

Rule Path
Disallow /

camcrawler

Rule Path
Disallow /

camelstampede

Rule Path
Disallow /

canon-webrecord

Rule Path
Disallow /

canon-webrecordpro

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cerberian drtrs

Rule Path
Disallow /

chameleon

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

check&get

Rule Path
Disallow /

checklinks

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

checklinks

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

chemiede-nodebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite

Rule Path
Disallow /

cherrypickerse

Rule Path
Disallow /

chilkat

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

cipinetbot

Rule Path
Disallow /

clariabot

Rule Path
Disallow /

climate ark

Rule Path
Disallow /

climateark spider

Rule Path
Disallow /

clushbot

Rule Path
Disallow /

collapsarweb

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

kinza

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mb2345browser

Rule Path
Disallow /

micromessenger

Rule Path
Disallow /

oppo\sa33

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

abot

Rule Path
Disallow /

accoona

Rule Path
Disallow /

acquia-crawler

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

adminshop.com

Rule Path
Disallow /

aggregator:vocus

Rule Path
Disallow /

ah-ha.com crawler

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

antibot-

Rule Path
Disallow /

appid: s~stremor-crawler-

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

aria equalizer

Rule Path
Disallow /

arianna.libero.it

Rule Path
Disallow /

asterias

Rule Path
Disallow /

athenusbot

Rule Path
Disallow /

attach

Rule Path
Disallow /

attrakt

Rule Path
Disallow /

attributor

Rule Path
Disallow /

augurfind

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

autowebdir

Rule Path
Disallow /

axfeedsbot

Rule Path
Disallow /

b2w

Rule Path
Disallow /

b2w/0.1

Rule Path
Disallow /

baypup

Rule Path
Disallow /

besserscheitern-crawl

Rule Path
Disallow /

betabot

Rule Path
Disallow /

bintellibot

Rule Path
Disallow /

bitlybot

Rule Path
Disallow /

bladder fusion

Rule Path
Disallow /

bnf.fr_bot

Rule Path
Disallow /

boitho.com-dc

Rule Path
Disallow /

bosug

Rule Path
Disallow /

bot

Rule Path
Disallow /

bottybot

Rule Path
Disallow /

btbot

Rule Path
Disallow /

bumblebee

Rule Path
Disallow /

byindia

Rule Path
Disallow /

byteserver

Rule Path
Disallow /

bzbot

Rule Path
Disallow /

c r a w l 3 r

Rule Path
Disallow /

cataguru

Rule Path
Disallow /

ccubee

Rule Path
Disallow /

cfetch

Rule Path
Disallow /

cis455crawler

Rule Path
Disallow /

citeseerxbot

Rule Path
Disallow /

cizilla.com/cizilla-

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

spaziodati

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10.0
Back to top

Warnings

  • 16 invalid lines.
Back to top