Log In Sign Up

amafilms.gr
robots.txt

Robots Exclusion Standard data for amafilms.gr

Resource Scan

Scan Details

Site Domain amafilms.gr
Base Domain amafilms.gr
Scan Status Ok
Last Scan2024-05-28T04:39:39+00:00
Next Scan 2024-06-27T04:39:39+00:00

Last Scan

Scanned2024-05-28T04:39:39+00:00
URL https://amafilms.gr/robots.txt
Redirect https://www.amafilms.gr/robots.txt
Redirect Domain www.amafilms.gr
Redirect Base amafilms.gr
Domain IPs 185.29.24.6
Redirect IPs 185.29.24.6
Response IP 185.29.24.6
Found Yes
Hash 2abcf775d50066f7fc550a09919b5accaff9f25573e00a07e613f362815a23a8
SimHash c14422ba17d8

Groups

aboundex/0.3

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

baiduspider-render/2.0

Rule Path
Disallow /

bigbozz/2.2.1

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cliqzbot/1.0

Rule Path
Disallow /

coccocbot-image/1.0

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

daum/

Rule Path
Disallow /

daum

Rule Path
Disallow /

daumoa 4.0

Rule Path
Disallow /

deusu

Rule Path
Disallow /

docomo/2.0

Rule Path
Disallow /

duckduckgo-favicons-bot/1.0

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

exploratodo/1.0

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

findxbot/1.0

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

german wikipedia broken weblinks bot

Rule Path
Disallow /

gigablastopensource/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

gimme60bot

Rule Path
Disallow /

gimmeusabot/1.0

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

lplinkcheck/nutch-1.12

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

metaspinner/1.0

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

odp

Rule Path
Disallow /

openfosbot

Rule Path
Disallow /

plukkie/1.6

Rule Path
Disallow /

psbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

safesearch microdata crawler

Rule Path
Disallow /

search.daum

Rule Path
Disallow /

seeker v.1

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

socialsearcher

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou pic spider/3.0

Rule Path
Disallow /

sogou web spider/3.0

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

sufog/nutch-2.2.1

Rule Path
Disallow /

tineye-bot/0.51

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

wbsearchbot/1.1

Rule Path
Disallow /

wbsrch/1.1

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

y!j-asr/0.1 crawler

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yandexmobilebot/3.0

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yellowpages.com

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

checks.panopta.com

Rule Path
Disallow /

freewebmonitoring sitechecker/0.1

Rule Path
Disallow /

hetrixtools uptime monitoring bot

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

montastic-monitor

Rule Path
Disallow /

nagios-plugins

Rule Path
Disallow /

notifyninja.com

Rule Path
Disallow /

pingdom.com_bot_version_1.4

Rule Path
Disallow /

sitemeer

Rule Path
Disallow /

siteuptime

Rule Path
Disallow /

status cake

Rule Path
Disallow /

uptimebot/1.0

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

uptime.com*

Rule Path
Disallow /

aportcatalogrobot/2.0

Rule Path
Disallow /

bot-pge.chlooe.com

Rule Path
Disallow /

ezoomsdisallow: /
feosey mohk crawler

Rule Path
Disallow /

fsb.php

Rule Path
Disallow /

hrcrawler/2.0

Rule Path
Disallow /

melvil rawi/1.0

Rule Path
Disallow /

mon_httpdownload

Rule Path
Disallow /

msie or firefox mutant

Rule Path
Disallow /

ocrawler

Rule Path
Disallow /

owler/0.4

Rule Path
Disallow /

redback

Rule Path
Disallow /

roboto

Rule Path
Disallow /

scanbot

Rule Path
Disallow /

science traveller international 1x/1.0

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

searchie/1.0

Rule Path
Disallow /

stratagems kumo

Rule Path
Disallow /

superpagesurlverifybot/1.0

Rule Path
Disallow /

testcrawler

Rule Path
Disallow /

thumbtack-thunderdome

Rule Path
Disallow /

truebot/1.0

Rule Path
Disallow /

user-agent

Rule Path
Disallow /

webhistorybot

Rule Path
Disallow /

wonderbot/js

Rule Path
Disallow /

vidiblescraper

Rule Path
Disallow /

vericitecrawler

Rule Path
Disallow /

venuscrawler

Rule Path
Disallow /

venuscrawler/nutch-1.12 (crawler@mycompany.com)

Rule Path
Disallow /

semalt.com

Rule Path
Disallow /

fix-website-errors.com

Rule Path
Disallow /

keywords-monitoring-your-success.com

Rule Path
Disallow /

traackr.com

Rule Path
Disallow /

seo-2-0.com

Rule Path
Disallow /

buttons-for-website.com

Rule Path
Disallow /

generic-pills-online.com

Rule Path
Disallow /

jonyx.net

Rule Path
Disallow /

freevpn.ninja

Rule Path
Disallow /

confib.ifmo.ru

Rule Path
Disallow /

mydc.net

Rule Path
Disallow /

vadimkravtcov.ru

Rule Path
Disallow /

vykup-avto-krasnodar.ru

Rule Path
Disallow /

writed.net

Rule Path
Disallow /

stairliftsarea.com

Rule Path
Disallow /

purchasepillsnorx.com

Rule Path
Disallow /

kinocccp.net

Rule Path
Disallow /

buynorxx.com

Rule Path
Disallow /

first-blog-grief-recovery.info

Rule Path
Disallow /

szukaj.onet.pl

Rule Path
Disallow /

dengi-pod-zalog-nedvizhimosti.ru

Rule Path
Disallow /

indiakino.net

Rule Path
Disallow /

larutti.ru

Rule Path
Disallow /

fb3.org

Rule Path
Disallow /

paxhosting.com

Rule Path
Disallow /

buynorxx.com

Rule Path
Disallow /

www.asseenontvonline.ru

Rule Path
Disallow /

www.newtechspb.ru

Rule Path
Disallow /

yourdesires.ru

Rule Path
Disallow /

light.ifmo.ru

Rule Path
Disallow /

pk-services.ru

Rule Path
Disallow /

lvov.mistr-x.org

Rule Path
Disallow /

musicstock.me

Rule Path
Disallow /

sarf3omlat.com

Rule Path
Disallow /

voloomoney.com

Rule Path
Disallow /

www.asseenontvonline.ru

Rule Path
Disallow /

nalogovyy-kodeks.ru

Rule Path
Disallow /

aktivator-windows10.blogspot.com

Rule Path
Disallow /

chelnytruck.ru

Rule Path
Disallow /

cpabegins.ru

Rule Path
Disallow /

gotomontenegro.net

Rule Path
Disallow /

rukino.org

Rule Path
Disallow /

www.lflash.ru

Rule Path
Disallow /

mypharmacydrugs.com

Rule Path
Disallow /

oxford-book.com.ua

Rule Path
Disallow /

mydirtystuff.com

Rule Path
Disallow /

telegraf.by

Rule Path
Disallow /

telegraf.ru

Rule Path
Disallow /

www.souvenirua.com

Rule Path
Disallow /

watchdogs-2.ru

Rule Path
Disallow /

oxford-book.com.ua

Rule Path
Disallow /

kursy-ege.ru

Rule Path
Disallow /

freetangodownload.com

Rule Path
Disallow /

timer4web.com

Rule Path
Disallow /

top10-way.com

Rule Path
Disallow /

w3data.co

Rule Path
Disallow /

domain2008.com

Rule Path
Disallow /

share-buttons-for-free.com

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

activesearchresults.com

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

attentio/nutch

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

biggerbetter

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bpimagewalker

Rule Path
Disallow /

bpimagewalker/2.0

Rule Path
Disallow /

barkrowler/0.7

Rule Path
Disallow /

bdcbot/1.0

Rule Path
Disallow /

brodie/1.0

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buck/2.2

Rule Path
Disallow /

businessseek.biz_spider

Rule Path
Disallow /

buzzbot/1.0

Rule Path
Disallow /

buzzsumo

Rule Path
Disallow /

calculon spider

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

catexplorador

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ccbot/1.0

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

changedetection.com

Rule Path
Disallow /

cipacrawler

Rule Path
Disallow /

citygridmedia/1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

cms crawler

Rule Path
Disallow /

cmscrawler

Rule Path
Disallow /

cognitiveseo

Rule Path
Disallow /

collection@infegy.com

Rule Path
Disallow /

commoncrawler node

Rule Path
Disallow /

companybook-crawler

Rule Path
Disallow /

contextad bot

Rule Path
Disallow /

contacts crawler

Rule Path
Disallow /

copubbot

Rule Path
Disallow /

corax

Rule Path
Disallow /

crawler@fast.no

Rule Path
Disallow /

crawlera/

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

css certificate spider

Rule Path
Disallow /

cybeye

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

datasift.com/bot

Rule Path
Disallow /

dbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

digincore

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

domaincheck.io crawler

Rule Path
Disallow /

domaincrawler/3.0

Rule Path
Disallow /

domainmacrocrawler/0.1

Rule Path
Disallow /

domainsigma.com/whois

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

ecairn-grabber/1.0

Rule Path
Disallow /

econtext/1.0

Rule Path
Disallow /

electricmonk/3.2.0

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

elmer, the thinglink imagebot

Rule Path
Disallow /

embedly

Rule Path
Disallow /

etoolsbot/1.0

Rule Path
Disallow /

everyonesocialbot/1.0

Rule Path
Disallow /

exabot

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

extlinksbot/1.5

Rule Path
Disallow /

factset research systems

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

faveeo/1.0

Rule Path
Disallow /

feedchecker-zocle

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

genieo

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

glindahl-cocrawler

Rule Path
Disallow /

gluten free crawler/1.0

Rule Path
Disallow /

golabbot

Rule Path
Disallow /

grammarly/1.0

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

grouphigh/1.0

Rule Path
Disallow /

hubspot webcrawler

Rule Path
Disallow /

hubspot links crawler

Rule Path
Disallow /

hubspot connect

Rule Path
Disallow /

http://search.thunderstone.com/texis/websearch/about.html

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

idbot

Rule Path
Disallow /

https://www.inboundlinks.win/inboundlinks/

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

infegy

Rule Path
Disallow /

infegyatlas/1.0

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

it2media-domain-crawler

Rule Path
Disallow /

james

Rule Path
Disallow /

kraken/0.1

Rule Path
Disallow /

kdslbot

Rule Path
Disallow /

lightspeedsystemscrawler

Rule Path
Disallow /

linkcheck by siteimprove.com

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkisbot/1.0

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

linkwalker/3.0

Rule Path
Disallow /

lipperhey-kaus-australis/5.0

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

livelapbot/0.2

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

ltbot/0.3.6

Rule Path
Disallow /

macinroy

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

magpie-crawler/1.1

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

mediametric

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

megaindex.com/2.0

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

merchantcentricbot

Rule Path
Disallow /

mixbot

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.7

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

mxt/nutch-1.12-snapshot

Rule Path
Disallow /

muckrackfeedparser

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

netcraft

Rule Path
Disallow /

netestate foaf crawler

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

netestate

Rule Path
Disallow /

netlyzer fastprobe

Rule Path
Disallow /

netseer

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

newsharecounts.com

Rule Path
Disallow /

nlpproject.info

Rule Path
Disallow /

nsrbot/1.0

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

online-webceo-bot

Rule Path
Disallow /

onpagebot

Rule Path
Disallow /

openwebspider

Rule Path
Disallow /

opendi screenshot bot

Rule Path
Disallow /

orgprobe/0.9.4

Rule Path
Disallow /

outclicksbot

Rule Path
Disallow /

pagefreezer

Rule Path
Disallow /

pagesinventory

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

pdffillerbot/1.0

Rule Path
Disallow /

percolatecrawler

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

pocketparser/2.0

Rule Path
Disallow /

popscreen bot

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

privacyawarebot

Rule Path
Disallow /

privatesearch

Rule Path
Disallow /

probethenet.com scanner

Rule Path
Disallow /

profound.net

Rule Path
Disallow /

prodvigatorbot/0.1

Rule Path
Disallow /

prospectb2b/2.0

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pu_in crawler

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

ravencrawler

Rule Path
Disallow /

rbot.info

Rule Path
Disallow /

rebelmouse

Rule Path
Disallow /

redditbot

Rule Path
Disallow /

replazbot

Rule Path
Disallow /

robotschecker/0.5

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

rssgraffiti

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

salesintelligent

Rule Path
Disallow /

scfcrawler

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

scrapyproject

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

scripted.com

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semalt.com

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot/6~bl

Rule Path
Disallow /

semrushbot/2~bl

Rule Path
Disallow /

semrushbot/1.2~bl

Rule Path
Disallow /

semrushbot/1.1~bl

Rule Path
Disallow /

semrushbot/3~bl

Rule Path
Disallow /

semrushbot/0.98~bl

Rule Path
Disallow /

semrushbot/1~bl

Rule Path
Disallow /

semrushbot/1.0~bl

Rule Path
Disallow /

semrushbot/0.99~bl

Rule Path
Disallow /

seodiver/1.0

Rule Path
Disallow /

seoengbot

Rule Path
Disallow /

seoengworldbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seologies/0.1

Rule Path
Disallow /

seolyticscrawler

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

seostats

Rule Path
Disallow /

shareaholicbot

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

simplepie

Rule Path
Disallow /

simplereach/1.1

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

siteanalyzerbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

siteimprove.com

Rule Path
Disallow /

slack-imgproxy

Rule Path
Disallow /

slackbot-linkexpanding

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

smabblerbot

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

socialmediascanner.eset.com

Rule Path
Disallow /

socialradarbot

Rule Path
Disallow /

socialrankiobot

Rule Path
Disallow /

socialsearcher

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

sourcescrub.com

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spiderbook

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

spidertest

Rule Path
Disallow /

spin3r

Rule Path
Disallow /

spotinfluence

Rule Path
Disallow /

springshare link checker

Rule Path
Disallow /

ssearch_bot

Rule Path
Disallow /

startmebot/1.0

Rule Path
Disallow /

steeler

Rule Path
Disallow /

steeler/3.5

Rule Path
Disallow /

storygizebot

Rule Path
Disallow /

surdotlybot/1.0

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

sysomos

Rule Path
Disallow /

teeraidbot

Rule Path
Disallow /

telesphoreo

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

thumbshots-de-bot

Rule Path
Disallow /

traackr.com

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

tsolcrawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tweetedtimes

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

twitterfeed 3

Rule Path
Disallow /

um-ln/1.0

Rule Path
Disallow /

umbot-ln/1.0

Rule Path
Disallow /

unwindfetchor

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

webmastercoffee

Rule Path
Disallow /

websays

Rule Path
Disallow /

webtarantula.com

Rule Path
Disallow /

wesee

Rule Path
Disallow /

whois.domaintools.com

Rule Path
Disallow /

wikido

Rule Path
Disallow /

worldbrewbot/2.1

Rule Path
Disallow /

wscheck

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yak/1.0

Rule Path
Disallow /

ziba/ziba-1.0.0

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

subscriber.zoominfo.com/zoominfo

Rule Path
Disallow /

appengine-google

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

apache-httpclient/4.5.2

Rule Path
Disallow /

apache-httpclient/4.5.6

Rule Path
Disallow /

camo asset proxy

Rule Path
Disallow /

commons-httpclient

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

curl

Rule Path
Disallow /

curl/7.29.0

Rule Path
Disallow /

curl/7.47.0

Rule Path
Disallow /

curl/7.15.0

Rule Path
Disallow /

curl/7.19.0

Rule Path
Disallow /

curl/7.29.0

Rule Path
Disallow /

curl/7.35.0

Rule Path
Disallow /

curl/7.42.1

Rule Path
Disallow /

curl/7.47.0

Rule Path
Disallow /

curl/7.54.0

Rule Path
Disallow /

curl/7.71.0

Rule Path
Disallow /

curl/7.71.1

Rule Path
Disallow /

curl/7.70.0

Rule Path
Disallow /

curl/7.69.0

Rule Path
Disallow /

curl/php 5.5.9-1ubuntu4.13

Rule Path
Disallow /

econtext/1.0

Rule Path
Disallow /

embeddedwb

Rule Path
Disallow /

fasthttp

Rule Path
Disallow /

gocrawl

Rule Path
Disallow /

go http package

Rule Path
Disallow /

go 1.1 package http

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

go-http-client/2.0

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

gocrawl

Rule Path
Disallow /

google-http-java-client

Rule Path
Disallow /

guzzlehttp

Rule Path
Disallow /

headlesschrome

Rule Path
Disallow /

headmasterseo

Rule Path
Disallow /

httpbin.org

Rule Path
Disallow /

http_request2

Rule Path
Disallow /

httpurlconnection

Rule Path
Disallow /

java

Rule Path
Disallow /

jersey

Rule Path
Disallow /

jersey/2.6

Rule Path
Disallow /

jersey/2.25.1

Rule Path
Disallow /

jersey/2.12

Rule Path
Disallow /

jersey/2.26

Rule Path
Disallow /

jersey/2.25.1

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

lua-resty-http

Rule Path
Disallow /

lwp::simple

Rule Path
Disallow /

matlab r2013a

Rule Path
Disallow /

mechanize/2.7.2

Rule Path
Disallow /

metainspector

Rule Path
Disallow /

metauri

Rule Path
Disallow /

mojolicious \(perl\)

Rule Path
Disallow /

nghttp2

Rule Path
Disallow /

nutch

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

pcore-http

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

psv3 (ssl connection)

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python

Rule Path
Disallow /

python/3.7 aiohttp/3.6.2

Rule Path
Disallow /

python-requests/2.21.0

Rule Path
Disallow /

python-requests/2.22.0

Rule Path
Disallow /

python-requests/2.23.0

Rule Path
Disallow /

python-requests/2.24.0

Rule Path
Disallow /

python-requests/2.25.0

Rule Path
Disallow /

python-requests/2.18.0

Rule Path
Disallow /

python-requests/2.18.1

Rule Path
Disallow /

python-requests/2.18.2

Rule Path
Disallow /

python-requests/2.18.3

Rule Path
Disallow /

python-requests/2.18.4

Rule Path
Disallow /

python-urllib/3.8

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

ruby

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

scrapy-redis

Rule Path
Disallow /

sitecrawler

Rule Path
Disallow /

symfony browserkit

Rule Path
Disallow /

synapse

Rule Path
Disallow /

wget/1.14

Rule Path
Disallow /

wget/1.20

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

www-mechanize

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

xml-rpc.net

Rule Path
Disallow /

zgrab/0.x

Rule Path
Disallow /

zoomspider

Rule Path
Disallow /

auto spider 1.0

Rule Path
Disallow /

babya discoverer

Rule Path
Disallow /

crawlera

Rule Path
Disallow /

crowsnest/0.5

Rule Path
Disallow /

datacha0s

Rule Path
Disallow /

domainsonocrawler/0.1

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

downloadbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

heritrix/3.1.1

Rule Path
Disallow /

heritrix/3.3.0

Rule Path
Disallow /

heritrix/3.3.0-snapshot-20150302-2206

Rule Path
Disallow /

jce

Rule Path
Disallow /

morfeus f scanner

Rule Path
Disallow /

mozilla/0.6 beta (windows)

Rule Path
Disallow /

mozilla-1.1

Rule Path
Disallow /

mozlila/5.0

Rule Path
Disallow /

msproxy/2.0

Rule Path
Disallow /

null

Rule Path
Disallow /

proxyjudge.info

Rule Path
Disallow /

proxyjudge

Rule Path
Disallow /

securityresearch.bot

Rule Path
Disallow /

sitechecker/0.1

Rule Path
Disallow /

wbsearchbot/1.1

Rule Path
Disallow /

webfuck

Rule Path
Disallow /

windows live writer

Rule Path
Disallow /

winhttp.winhttprequest.5

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

zollard

Rule Path
Disallow /

comodo-webinspector-crawler

Rule Path
Disallow /

erratasecscanner

Rule Path
Disallow /

httpscheck

Rule Path
Disallow /

load impact

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

masscan

Rule Path
Disallow /

mfibot/1.1

Rule Path
Disallow /

muhstik-scan

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

nmap scripting engine

Rule Path
Disallow /

nmap

Rule Path
Disallow /

project25499.com

Rule Path
Disallow /

proxytest.zmap.io

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

riddler

Rule Path
Disallow /

safesearch microdata crawler

Rule Path
Disallow /

scanalert

Rule Path
Disallow /

scan.nextcloud.com

Rule Path
Disallow /

scanning for research

Rule Path
Disallow /

sucuri.net

Rule Path
Disallow /

sitelock

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

ssl labs

Rule Path
Disallow /

sysscan

Rule Path
Disallow /

virustotalcloud

Rule Path
Disallow /

zgrab/0.x

Rule Path
Disallow /

microsoft office/14.0

Rule Path
Disallow /

microsoft office/15.0

Rule Path
Disallow /

microsoft office/16.0

Rule Path
Disallow /

skype for business

Rule Path
Disallow /

openoffice/4.1.1

Rule Path
Disallow /

cpanel-http-client/1.0

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/

Other Records

Field Value
crawl-delay 30
Back to top

Warnings

  • 30 invalid lines.
Back to top