Log In Sign Up

cronicasdeunpueblo.es
robots.txt

Robots Exclusion Standard data for cronicasdeunpueblo.es

Resource Scan

Scan Details

Site Domain cronicasdeunpueblo.es
Base Domain cronicasdeunpueblo.es
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-21T07:19:14+00:00
Next Scan 2025-01-19T07:19:14+00:00

Last Successful Scan

Scanned2023-04-01T00:42:02+00:00
URL https://cronicasdeunpueblo.es/robots.txt
Domain IPs 104.21.87.91, 172.67.169.79, 2606:4700:3030::6815:575b, 2606:4700:3037::ac43:a94f
Response IP 172.67.169.79
Found Yes
Hash 35a11ea2750a3ca3075d30976538c2359165ce88bf340e6adecb03c7c06521be
SimHash 12465160c4ea

Groups

buck

Rule Path
Disallow /

twitterbot

Rule Path
Disallow *
Allow /upload

moreover

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

moodlebot/1.0

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot/0.1

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

nutch

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

linkdex.com/v2.0

Rule Path
Disallow /

flamingo_searchengine+(+http://www.flamingosearch.com/bot)

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

owlin bot v. 3.0

Rule Path
Disallow /

owlin bot v3

Rule Path
Disallow /

owlin bot

Rule Path
Disallow /

owlin

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

*

Rule Path
Disallow /app
Disallow /timeout

yoozbot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 120
Back to top

Warnings

  • 10 invalid lines.
Back to top