agerpres.ro robots.txt

*

Rule	Path
Disallow	/*.rss$
Disallow	/*.png$
Disallow	/*/page/
Disallow	/webmagnat/
Disallow	/articol/get_page_views*
Disallow	/articol/get_page_views
Disallow	/acasa/flux
Disallow	/facebook_hook/most_commented/*
Disallow	/acasa/flux*
Disallow	/flux
Disallow	/login
Disallow	/?is_search=
Disallow	/categorie?is_search=*
Disallow	/notificari/getAjaxTotal
Disallow	/*?is_search=
Disallow	/?is_search=
Disallow	/categorie?is_search=
Disallow	/notificari/getAjaxTotal*
Disallow	/notificari/getAjaxTotal
Disallow	/foto/watermarkList/*
Disallow	/foto/watermarkList/
Disallow	/latest_news/interne
Disallow	/latest_news/externe
Disallow	/latest_news/politica-externa
Disallow	/latest_news/
Disallow	/latest_news/*
Disallow	*/acasa/carousel
Disallow	/acasa/carousel

Rule

Path

Disallow

/*.rss$

Disallow

/*.png$

Disallow

/*/page/

Disallow

/webmagnat/

Disallow

/articol/get_page_views*

Disallow

/articol/get_page_views

Disallow

/acasa/flux

Disallow

/facebook_hook/most_commented/*

Disallow

/acasa/flux*

Disallow

/flux

Disallow

/login

Disallow

/*?is_search=*

Disallow

/categorie?is_search=*

Disallow

/notificari/getAjaxTotal

Disallow

/*?is_search=

Disallow

/?is_search=

Disallow

/categorie?is_search=

Disallow

/notificari/getAjaxTotal*

Disallow

*/notificari/getAjaxTotal*

Disallow

/foto/watermarkList/*

Disallow

/foto/watermarkList/

Disallow

/latest_news/interne

Disallow

/latest_news/externe

Disallow

/latest_news/politica-externa

Disallow

/latest_news/

Disallow

/latest_news/*

Disallow

*/acasa/carousel

Disallow

/acasa/carousel

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

semrushbot-ba

Rule	Path
Disallow	/

Rule

Path

Disallow

/

semrushbot-sa

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aspiegelbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

semanticscholarbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

adldxbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

bingbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

bingbot/2.0

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

bingpreview/1.0b

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

bingpreview

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

msnbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

msnbot-media

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

applebot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

cxensebot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

cxensebot/2.0

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

petalbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

jeeves
teoma

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

slurp

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

duckduckbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

baiduspider

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

yandex

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

yandexbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

ia_archiver

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

archive.org_bot

Rule	Path
Allow	/foto/watermarkList/*
Allow	/foto/watermarkList/
Allow	*/acasa/carousel
Allow	/acasa/carousel

Rule

Path

Allow

/foto/watermarkList/*

Allow

/foto/watermarkList/

Allow

*/acasa/carousel

Allow

/acasa/carousel

Other Records

Field	Value
crawl-delay	25

Field

Value

crawl-delay

25

bot\*

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

mj12bot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

blog

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

dotbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

obot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

blexbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

storebot-google

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

googlebot-news

Rule	Path
Allow	*/acasa/carousel
Allow	/acasa/carousel

Rule

Path

Allow

*/acasa/carousel

Allow

/acasa/carousel

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

10

googlebot-video

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

googlebot-image

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

mediapartners-google

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

duplexweb-google

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	35

Field

Value

crawl-delay

35

googlebot

Rule	Path
Allow	*/acasa/carousel
Allow	/acasa/carousel

Rule

Path

Allow

*/acasa/carousel

Allow

/acasa/carousel

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

5

*

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	45

Field

Value

crawl-delay

45

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alittle client

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aspseek

Rule

Path

Disallow

/

abonti

Rule

Path

Disallow

/

aboundex

Rule

Path

Disallow

/

aboundexbot

Rule

Path

Disallow

/

acunetix

Rule

Path

Disallow

/

afd-verbotsverfahren

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

aihitbot

Rule

Path

Disallow

/

aipbot

Rule

Path

Disallow

/

alexibot

Rule

Path

Disallow

/

allsubmitter

Rule

Path

Disallow

/

alligator

Rule

Path

Disallow

/

alphabot

Rule

Path

Disallow

/

anarchie

Rule

Path

Disallow

/

anarchy

Rule

Path

Disallow

/

anarchy99

Rule

Path

Disallow

/

ankit

Rule

Path

Disallow

/

anthill

Rule

Path

Disallow

/

apexoo

Rule

Path

Disallow

/

aspiegel

Rule

Path

Disallow

/

asterias

Rule

Path

Disallow

/

atomseobot

Rule

Path

Disallow

/

attach

Rule

Path

Disallow

/

awariorssbot

Rule

Path

Disallow

/

awariosmartbot

Rule

Path

Disallow

/

bbbike

Rule

Path

Disallow

/

bdcbot

Rule

Path

Disallow

/

bdfetch

Rule

Path

Disallow

/

blexbot

Rule

Path

Disallow

/

backdoorbot

Rule

Path

Disallow

/

backstreet

Rule

Path

Disallow

/

backweb

Rule

Path

Disallow

/

backlink-ceck

Rule

Path

Disallow

/

backlinkcrawler

Rule

Path

Disallow

/

badass

Rule

Path

Disallow

/

bandit

Rule

Path

Disallow

/

barkrowler

Rule

Path

Disallow

/

batchftp

Rule

Path

Disallow

/

battleztar bazinga

Rule

Path

Disallow

/

betabot

Rule

Path

Disallow

/

bigfoot

Rule

Path

Disallow

/

bitacle

Rule

Path

Disallow

/

black hole

Rule

Path

Disallow

/

blackwidow

Rule

Path

Disallow

/

blackboard

Rule

Path

Disallow

/

blow

Rule

Path

Disallow

/

blowfish

Rule

Path

Disallow

/

boardreader

Rule

Path

Disallow

/

bolt

Rule

Path

Disallow

/

botalot

Rule

Path

Disallow

/

brandprotect

Rule

Path

Disallow

/

brandwatch

Rule

Path

Disallow

/

buck

Rule

Path

Disallow

/

buddy

Rule

Path

Disallow

/

builtbottough

Rule

Path

Disallow

/

builtwith

Rule

Path

Disallow

/

bullseye

Rule

Path

Disallow

/

bunnyslippers

Rule

Path

Disallow

/

buzzsumo

Rule

Path

Disallow

/

catexplorador

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

code87

Rule

Path

Disallow

/

cshttp

Rule

Path

Disallow

/

calculon

Rule

Path

Disallow

/

cazoodlebot

Rule

Path

Disallow

/

cegbfeieh

Rule

Path

Disallow

/

censysinspect

Rule

Path

Disallow

/

cheteam

Rule

Path

Disallow

/

cheesebot

Rule

Path

Disallow

/

cherrypicker

Rule

Path

Disallow

/

chinaclaw

Rule

Path

Disallow

/

chlooe

Rule

Path

Disallow

/

citoid

Rule

Path

Disallow

/

claritybot

Rule

Path

Disallow

/

cliqzbot

Rule

Path

Disallow

/

cloud mapping

Rule

Path

Disallow

/

cocolyzebot

Rule

Path

Disallow

/

cogentbot

Rule

Path

Disallow

/

collector

Rule

Path

Disallow

/

copier

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

copyscape

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

craftbot

Rule

Path

Disallow

/

crawling at home project

Rule

Path

Disallow

/

crazywebcrawler

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crunchbot

Rule

Path

Disallow

/

curious

Rule

Path

Disallow

/

custo

Rule

Path

Disallow

/

cyotekwebcopy

Rule

Path

Disallow

/

dblbot

Rule

Path

Disallow

/

diibot

Rule

Path

Disallow

/

dsearch

Rule

Path

Disallow

/

dts agent

Rule

Path

Disallow

/

datacha0s

Rule

Path

Disallow

/

databasedrivermysqli

Rule

Path

Disallow

/

demon

Rule

Path

Disallow

/

deusu

Rule

Path

Disallow

/

devil

Rule

Path

Disallow

/

digincore

Rule

Path

Disallow

/

digitalpebble

Rule

Path

Disallow

/

dirbuster

Rule

Path

Disallow

/

disco

Rule

Path

Disallow

/

discobot

Rule

Path

Disallow

/

discoverybot

Rule

Path

Disallow

/

dispatch

Rule

Path

Disallow

/

dittospyder

Rule

Path

Disallow

/

dnbcrawler-analytics

Rule

Path

Disallow

/

dnyzbot

Rule

Path

Disallow

/

domcopbot

Rule

Path

Disallow

/

domainappender

Rule

Path

Disallow

/

domaincrawler

Rule

Path

Disallow

/

domainsigmacrawler

Rule

Path

Disallow

/

domainstatsbot

Rule

Path

Disallow

/

domains project

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

download wonder

Rule

Path

Disallow

/

dragonfly

Rule

Path

Disallow

/

drip

Rule

Path

Disallow

/

eccp/1.0

Rule

Path

Disallow

/

email siphon

Rule

Path

Disallow

/

email wolf

Rule

Path

Disallow

/

easydl

Rule

Path

Disallow

/

ebingbong

Rule

Path

Disallow

/

ecxi

Rule

Path

Disallow

/

eirgrabber

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

evil

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

express webpictures

Rule

Path

Disallow

/

extlinksbot

Rule

Path

Disallow

/

extractor

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

extreme picture finder

Rule

Path

Disallow

/

eyenetie

Rule

Path

Disallow

/

ezooms

Rule

Path

Disallow

/

fdm

Rule

Path

Disallow

/

fhscan

Rule

Path

Disallow

/

femtosearchbot

Rule

Path

Disallow

/

fimap

Rule

Path

Disallow

/

firefox/7.0

Rule

Path

Disallow

/

flashget

Rule

Path

Disallow

/

flunky

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

freeuploader

Rule

Path

Disallow

/

frontpage

Rule

Path

Disallow

/

fuzz

Rule

Path

Disallow

/

fyberspider

Rule

Path

Disallow

/

fyrebot

Rule

Path

Disallow

/

g-i-g-a-b-o-t

Rule

Path

Disallow

/

gt::www

Rule

Path

Disallow

/

galaxybot

Rule

Path

Disallow

/

genieo

Rule

Path

Disallow

/

germcrawler

Rule

Path

Disallow

/

getright

Rule

Path

Disallow

/

getweb

Rule

Path

Disallow

/

getintent

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

go!zilla

Rule

Path

Disallow

/

go-ahead-got-it

Rule

Path

Disallow

/

gozilla

Rule

Path

Disallow

/

gotit

Rule

Path

Disallow

/

grabnet

Rule

Path

Disallow

/

grabber

Rule

Path

Disallow

/

grafula

Rule

Path

Disallow

/

grapefx

Rule

Path

Disallow

/

grapeshotcrawler

Rule

Path

Disallow

/

gridbot

Rule

Path

Disallow

/

headmasterseo

Rule

Path

Disallow

/

hmview

Rule

Path

Disallow

/

htmlparser

Rule

Path

Disallow

/

http::lite

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

haansoft

Rule

Path

Disallow

/

haosouspider

Rule

Path

Disallow

/

harvest

Rule

Path

Disallow

/

havij

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

honolulubot

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

hybridbot

Rule

Path

Disallow

/

idbte4m

Rule

Path

Disallow

/

idbot

Rule

Path

Disallow

/

irlbot

Rule

Path

Disallow

/

iblog

Rule

Path

Disallow

/

id-search

Rule

Path

Disallow

/

ilsebot

Rule

Path

Disallow

/

image fetch

Rule

Path

Disallow

/

image sucker

Rule

Path

Disallow

/

indeedbot

Rule

Path

Disallow

/

indy library

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

infotekies

Rule

Path

Disallow

/

intelliseek

Rule

Path

Disallow

/

interget

Rule

Path

Disallow

/

internet ninja

Rule

Path

Disallow

/

internetseer

Rule

Path

Disallow

/

iria

Rule

Path

Disallow

/

iskanie

Rule

Path

Disallow

/

istellabot

Rule

Path

Disallow

/

joc web spider

Rule

Path

Disallow

/

jamesbot

Rule

Path

Disallow

/

jbrofuzz

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

jetcar

Rule

Path

Disallow

/

jetty

Rule

Path

Disallow

/

jikespider

Rule

Path

Disallow

/

joomla

Rule

Path

Disallow

/

jorgee

Rule

Path

Disallow

/

justview

Rule

Path

Disallow

/

jyxobot

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

keybot translation-search-machine

Rule

Path

Disallow

/

keyword density

Rule

Path

Disallow

/

kinza

Rule

Path

Disallow

/

kozmosbot

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

lwp::simple

Rule

Path

Disallow

/

lanshanbot

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

leap

Rule

Path

Disallow

/

leechftp

Rule

Path

Disallow

/

leechget

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

lftp

Rule

Path

Disallow

/

libweb

Rule

Path

Disallow

/

libwhisker

Rule

Path

Disallow

/

liebaofast

Rule

Path

Disallow

/

lightspeedsystems

Rule

Path

Disallow

/

likse

Rule

Path

Disallow

/

linkscan

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

linkbot

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkpadbot

Rule

Path

Disallow

/

linksmanager

Rule

Path

Disallow

/

linqiametadatadownloaderbot

Rule

Path

Disallow

/

linqiarssbot

Rule

Path

Disallow

/

linqiascrapebot

Rule

Path

Disallow

/

lipperhey

Rule

Path

Disallow

/

lipperhey spider

Rule

Path

Disallow

/

litemage_walker

Rule

Path

Disallow

/

lmspider

Rule

Path

Disallow

/

ltx71

Rule

Path

Disallow

/

mfc_tear_sample

Rule

Path

Disallow

/

midown tool

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

mqqbrowser

Rule

Path

Disallow

/

msfrontpage

Rule

Path

Disallow

/

msiecrawler

Rule

Path

Disallow

/

mtrobot

Rule

Path

Disallow

/

mag-net

Rule

Path

Disallow

/

magnet

Rule

Path

Disallow

/

mail.ru_bot

Rule

Path

Disallow

/

majestic seo

Rule

Path

Disallow

/

majestic-seo

Rule

Path

Disallow

/

majestic12

Rule

Path

Disallow

/

markmonitor

Rule

Path

Disallow

/

markwatch

Rule

Path

Disallow

/

mass downloader

Rule

Path

Disallow

/

masscan

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

mauibot

Rule

Path

Disallow

/

mb2345browser

Rule

Path

Disallow

/

meanpath bot

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

mediatoolkitbot

Rule

Path

Disallow

/

megaindex.ru

Rule

Path

Disallow

/

metauri

Rule

Path

Disallow

/

micromessenger

Rule

Path

Disallow

/

microsoft data access

Rule

Path

Disallow

/

microsoft url control

Rule

Path

Disallow

/

minefield

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

moblie safari

Rule

Path

Disallow

/

mojeek

Rule

Path

Disallow

/

mojolicious

Rule

Path

Disallow

/

molokaibot

Rule

Path

Disallow

/

morfeus fucking scanner

Rule

Path

Disallow

/

mozlila

Rule

Path

Disallow

/

mr.4x3

Rule

Path

Disallow

/

msrabot

Rule

Path

Disallow

/

musobot

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

npbot

Rule

Path

Disallow

/

name intelligence

Rule

Path

Disallow

/

nameprotect

Rule

Path

Disallow

/

navroad

Rule

Path

Disallow

/

nearsite

Rule

Path

Disallow

/

needle

Rule

Path

Disallow

/

nessus

Rule

Path

Disallow

/

net vampire

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netlyzer

Rule

Path

Disallow

/

netmechanic

Rule

Path

Disallow

/

netspider

Rule

Path

Disallow

/

netzip

Rule

Path

Disallow

/

netcraft

Rule

Path

Disallow

/

nettrack

Rule

Path

Disallow

/

netvibes

Rule

Path

Disallow

/

nextgensearchbot

Rule

Path

Disallow

/

nibbler

Rule

Path

Disallow

/

niki-bot

Rule

Path

Disallow

/

nikto

Rule

Path

Disallow

/

nimblecrawler

Rule

Path

Disallow

/

nimbostratus

Rule

Path

Disallow

/

ninja

Rule

Path

Disallow

/

nmap

Rule

Path

Disallow

/

not

Rule

Path

Disallow

/

nuclei

Rule

Path

Disallow

/

nutch

Rule

Path

Disallow

/

octopus

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

offline navigator

Rule

Path

Disallow

/

oncrawl

Rule

Path

Disallow

/

openlinkprofiler

Rule

Path

Disallow

/

openvas

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openvas

Rule

Path

Disallow

/

orangebot

Rule

Path

Disallow

/

orangespider

Rule

Path

Disallow

/

outclicksbot

Rule

Path

Disallow

/

outfoxbot

Rule

Path

Disallow

/

pecl::http

Rule

Path

Disallow

/

phpcrawl

Rule

Path

Disallow

/

poe-component-client-http

Rule

Path

Disallow

/

page analyzer

Rule

Path

Disallow

/

pageanalyzer

Rule

Path

Disallow

/

pagegrabber

Rule

Path

Disallow

/

pagescorer

Rule

Path

Disallow

/

pagething.com

Rule

Path

Disallow

/

pandalytics

Rule

Path

Disallow

/

panscient

Rule

Path

Disallow

/

papa foto

Rule

Path

Disallow

/

pavuk

Rule

Path

Disallow

/

peoplepal

Rule

Path

Disallow

/

petalbot

Rule

Path

Disallow

/

pi-monster

Rule

Path

Disallow

/

picscout

Rule

Path

Disallow

/

picsearch

Rule

Path

Disallow

/

picturefinder

Rule

Path

Disallow

/

piepmatz

Rule

Path

Disallow

/

pimonster

Rule

Path

Disallow

/

pixray

Rule

Path

Disallow

/

pleasecrawl

Rule

Path

Disallow

/

pockey

Rule

Path

Disallow

/

propowerbot

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

probethenet

Rule

Path

Disallow

/

psbot

Rule

Path

Disallow

/

pu_in

Rule

Path

Disallow

/

pump

Rule

Path

Disallow

/

pxbroker

Rule

Path

Disallow

/

pycurl

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

quick-crawler

Rule

Path

Disallow

/

rssingbot

Rule

Path

Disallow

/

rankactive

Rule

Path

Disallow

/

rankactivelinkbot

Rule

Path

Disallow

/

rankflex

Rule

Path

Disallow

/

rankingbot

Rule

Path

Disallow

/

rankingbot2

Rule

Path

Disallow

/

rankivabot

Rule

Path

Disallow

/

rankurbot

Rule

Path

Disallow

/

re-re

Rule

Path

Disallow

/

reget

Rule

Path

Disallow

/

realdownload

Rule

Path

Disallow

/

reaper

Rule

Path

Disallow

/

rebelmouse

Rule

Path

Disallow

/

recorder

Rule

Path

Disallow

/

redesscrapy

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

ripper

Rule

Path

Disallow

/

rocketcrawler

Rule

Path

Disallow

/

rogerbot

Rule

Path

Disallow

/

sbider

Rule

Path

Disallow

/

seokicks

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seolyticscrawler

Rule

Path

Disallow

/

seoprofiler

Rule

Path

Disallow

/

seostats

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

smtbot

Rule

Path

Disallow

/

salesintelligent

Rule

Path

Disallow

/

scanalert

Rule

Path

Disallow

/

scanbot

Rule

Path

Disallow

/

scoutjet

Rule

Path

Disallow

/

scrapy

Rule

Path

Disallow

/

screaming

Rule

Path

Disallow

/

screenerbot

Rule

Path

Disallow

/

screpybot

Rule

Path

Disallow

/

searchestate

Rule

Path

Disallow

/

searchmetricsbot

Rule

Path

Disallow

/

seekport

Rule

Path

Disallow

/

semanticjuice

Rule

Path

Disallow

/

semrush

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

sentibot

Rule

Path

Disallow

/

seositecheckup

Rule

Path

Disallow

/

seobilitybot

Rule

Path

Disallow

/

seomoz

Rule

Path

Disallow

/

shodan

Rule

Path

Disallow

/

siphon

Rule

Path

Disallow

/

site sucker

Rule

Path

Disallow

/

sitecheckerbotcrawler

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

sitelockspider

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

sitesucker

Rule

Path

Disallow

/

sitebeam

Rule

Path

Disallow

/

siteimprove

Rule

Path

Disallow

/

sitevigil

Rule

Path

Disallow

/

slysearch

Rule

Path

Disallow

/

smartdownload

Rule

Path

Disallow

/

snake

Rule

Path

Disallow

/

snapbot

Rule

Path

Disallow

/

snoopy

Rule

Path

Disallow

/

socialrankiobot

Rule

Path

Disallow

/

sociscraper

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

sosospider

Rule

Path

Disallow

/

sottopop

Rule

Path

Disallow

/

spacebison

Rule

Path

Disallow

/

spammen

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

spinn3r

Rule

Path

Disallow

/

sputnikbot

Rule

Path

Disallow

/

sqlmap

Rule

Path

Disallow

/

sqlworm

Rule

Path

Disallow

/

sqworm

Rule

Path

Disallow

/

steeler

Rule

Path

Disallow

/

stripper

Rule

Path

Disallow

/

sucker

Rule

Path

Disallow

/

sucuri

Rule

Path

Disallow

/

superbot

Rule

Path

Disallow

/

superhttp

Rule

Path

Disallow

/

surfbot

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

swiftbot

Rule

Path

Disallow

/

szukacz

Rule

Path

Disallow

/

t0phackteam

Rule

Path

Disallow

/

t8abot

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

telesphoreo

Rule

Path

Disallow

/

telesphorep

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

thumbor

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

toata

Rule

Path

Disallow

/

toweyabot

Rule

Path

Disallow

/

tracemyfile

Rule

Path

Disallow

/

trendiction

Rule

Path

Disallow

/

trendictionbot

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

turnitin

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

twengabot

Rule

Path

Disallow

/

twice

Rule

Path

Disallow

/

typhoeus

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

urly.warning

Rule

Path

Disallow

/

unisterbot

Rule

Path

Disallow

/

upflow

Rule

Path

Disallow

/

v-bot

Rule

Path

Disallow

/

vb project

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vacuum

Rule

Path

Disallow

/

vagabondo

Rule

Path

Disallow

/

velenpublicwebcrawler

Rule

Path

Disallow

/

vericitecrawler

Rule

Path

Disallow

/

vidiblescraper

Rule

Path

Disallow

/

virusdie

Rule

Path

Disallow

/

voideye

Rule

Path

Disallow

/

voil

Rule

Path

Disallow

/

voltron

Rule

Path

Disallow

/

wasalive-bot

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

webdav

Rule

Path

Disallow

/

wisenutbot

Rule

Path

Disallow

/

wpscan

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

www-mechanize

Rule

Path

Disallow

/

www::mechanize

Rule

Path

Disallow

/

wwwoffle

Rule

Path

Disallow

/

wallpapers

Rule

Path

Disallow

/

wallpapers/3.0

Rule

Path

Disallow

/

wallpapershd

Rule

Path

Disallow

/

wesee

Rule

Path

Disallow

/

web auto

Rule

Path

Disallow

/

web collage

Rule

Path

Disallow

/

web enhancer

Rule

Path

Disallow

/

web fetch

Rule

Path

Disallow

/

web fuck

Rule

Path

Disallow

/

web pix

Rule

Path

Disallow

/

web sauger

Rule

Path

Disallow

/

web sucker

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webcollage

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

webfetch

Rule

Path

Disallow

/

webfuck

Rule

Path

Disallow

/

webgo is

Rule

Path

Disallow

/

webimagecollector

Rule

Path

Disallow

/

webleacher

Rule

Path

Disallow

/

webpix

Rule

Path

Disallow

/

webreaper

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

websucker

Rule

Path

Disallow

/

webwhacker

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webalta

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

webshag

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

websiteextractor

Rule

Path

Disallow

/

websitequester

Rule

Path

Disallow

/

webster

Rule

Path

Disallow

/

whack

Rule

Path

Disallow

/

whacker

Rule

Path

Disallow

/

whatweb

Rule

Path

Disallow

/

who.is bot

Rule

Path

Disallow

/

widow

Rule

Path

Disallow

/

winhttrack

Rule

Path

Disallow

/

wiseguys robot

Rule

Path

Disallow

/

wonderbot

Rule

Path

Disallow

/

woobot

Rule

Path

Disallow

/

wotbox

Rule

Path

Disallow

/

wprecon

Rule

Path

Disallow

/

xaldon webspider

Rule

Path

Disallow

/

xaldon_webspider

Rule

Path

Disallow

/

xenu

Rule

Path

Disallow

/

youdaobot

Rule

Path

Disallow

/

zade

Rule

Path

Disallow

/

zauba

Rule

Path

Disallow

/

zermelo

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zitebot

Rule

Path

Disallow

/

zmeu

Rule

Path

Disallow

/

zoombot

Rule

Path

Disallow

/

zoominfobot

Rule

Path

Disallow

/

zumbot

Rule

Path

Disallow

/

zyborg

Rule

Path

Disallow

/

adscanner

Rule

Path

Disallow

/

archive.org_bot

Rule

Path

Disallow

/

arquivo-web-crawler

Rule

Path

Disallow

/

arquivo.pt

Rule

Path

Disallow

/

autoemailspider

Rule

Path

Disallow

/

backlink-check

Rule

Path

Disallow

/

cah.io.community

Rule

Path

Disallow

/

check1.exe

Rule

Path

Disallow

/

clark-crawler

Rule

Path

Disallow

/

coccocbot

Rule

Path

Disallow

/

cognitiveseo

Rule

Path

Disallow

/

com.plumanalytics

Rule

Path

Disallow

/

crawl.sogou.com

Rule

Path

Disallow

/

crawler.feedback

Rule

Path

Disallow

/

crawler4j

Rule

Path

Disallow

/

dataforseo.com

Rule

Path

Disallow

/

demandbase-bot

Rule

Path

Disallow

/

domainsproject.org

Rule

Path

Disallow

/

ecatch

Rule

Path

Disallow

/

evc-batch

Rule

Path

Disallow

/

facebookscraper

Rule

Path

Disallow

/

gopher

Rule

Path

Disallow

/

heritrix

Rule

Path

Disallow

/

instabid

Rule

Path

Disallow

/

internetvista monitor

Rule

Path

Disallow

/

ips-agent

Rule

Path

Disallow

/

isitwp.com

Rule

Path

Disallow

/

iubenda-radar

Rule

Path

Disallow

/

linkdexbot

Rule

Path

Disallow

/

lwp-request

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

magpie-crawler

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

mediawords

Rule

Path

Disallow

/

muhstik-scan

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

obot

Rule

Path

Disallow

/

page scorer

Rule

Path

Disallow

/

pcbrowser

Rule

Path

Disallow

/

plumanalytics

Rule

Path

Disallow

/

polaris version

Rule

Path

Disallow

/

probe-image-size

Rule

Path

Disallow

/

ripz

Rule

Path

Disallow

/

s1z.ru

Rule

Path

Disallow

/

satoristudio.net

Rule

Path

Disallow

/

scalaj-http

Rule

Path

Disallow

/

scan.lol

Rule

Path

Disallow

/

seobility

Rule

Path

Disallow

/

seocompany.store

Rule

Path

Disallow

/

seoscanners

Rule

Path

Disallow

/

seostar

Rule

Path

Disallow

/

serpstatbot

Rule

Path

Disallow

/

sexsearcher

Rule

Path

Disallow

/

sitechecker.pro

Rule

Path

Disallow

/

siteripz

Rule

Path

Disallow

/

sogouspider

Rule

Path

Disallow

/

sp_auditbot

Rule

Path

Disallow

/

spyfu

Rule

Path

Disallow

/

sysscan

Rule

Path

Disallow

/

takeout

Rule

Path

Disallow

/

trendiction.com

Rule

Path

Disallow

/

trendiction.de

Rule

Path

Disallow

/

ubermetrics-technologies.com

Rule

Path

Disallow

/

voyagerx.com

Rule

Path

Disallow

/

webgains-bot

Rule

Path

Disallow

/

webmeup-crawler

Rule

Path

Disallow

/

webpros.com

Rule

Path

Disallow

/

webprosbot

Rule

Path

Disallow

/

x09mozilla

Rule

Path

Disallow

/

x22mozilla

Rule

Path

Disallow

/

xpymep1.exe

Rule

Path

Disallow

/

zauba.io

Rule

Path

Disallow

/

zgrab

Rule

Path

Disallow

/

israbot

Rule

Path

Disallow

/

orthogaffe

Rule

Path

Disallow

/

ubicrawler

Rule

Path

Disallow

/

doc

Rule

Path

Disallow

/

zao

Rule

Path

Disallow

/

sitecheck.internetseer.com

Rule

Path

Disallow

/

zealbot

Rule

Path

Disallow

/

msiecrawler

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

fetch

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

linko

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

microsoft.url.control

Rule

Path

Disallow

/

xenu

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

libwww

Rule

Path

Disallow

/

zyborg

Rule

Path

Disallow

/

download ninja

Rule

Path

Disallow

/

fast

Rule

Path

Disallow

/

grub-client

Rule

Path

Disallow

/

Site Domain	agerpres.ro
Base Domain	agerpres.ro
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-09-14T16:58:01+00:00
Next Scan	2024-12-13T16:58:01+00:00

Scanned	2022-11-19T18:47:24+00:00
URL	https://agerpres.ro/robots.txt
Response IP	104.21.31.17, 172.67.174.164
Found	Yes
Hash	352e5143aed00fc7fe6e15f41552bc0285b70d116d55f444fd08a2e4148709ca
SimHash	6b17677121af

agerpres.rorobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

ahrefsbot

semrushbot

semrushbot-ba

semrushbot-sa

aspiegelbot

semanticscholarbot

facebookexternalhit

Other Records

facebookexternalhit/1.1

Other Records

adldxbot

Other Records

bingbot

Other Records

bingbot/2.0

Other Records

bingpreview/1.0b

Other Records

bingpreview

Other Records

msnbot

Other Records

msnbot-media

Other Records

applebot

Other Records

cxensebot

Other Records

cxensebot/2.0

Other Records

petalbot

Other Records

jeevesteoma

Other Records

slurp

Other Records

duckduckbot

Other Records

baiduspider

Other Records

yandex

Other Records

yandexbot

Other Records

ia_archiver

Other Records

archive.org_bot

Other Records

bot\*

Other Records

mj12bot

Other Records

blog

Other Records

dotbot

Other Records

obot

Other Records

blexbot

Other Records

storebot-google

Other Records

googlebot-news

Other Records

googlebot-video

Other Records

googlebot-image

Other Records

mediapartners-google

Other Records

duplexweb-google

Other Records

googlebot

Other Records

agerpres.ro
robots.txt

jeeves
teoma