dietazas.com
robots.txt

Groups

*

googlebot-image

adsbot-google

googlebot-mobile

yandex

msiecrawler

webcopier

httrack

microsoft.url.control

libwww

noxtrumbot

No rules defined. All paths allowed.

Other Records

msnbot

No rules defined. All paths allowed.

Other Records

slurp

No rules defined. All paths allowed.

Other Records

ia_archiver disallow: /
addthis.com disallow: /
admantx disallow: /
ahrefsbot disallow: /
bdcbot disallow: /
bender disallow: /
bixocrawler disallow: /
bl.uk_lddc_bot disallow: /
blexbot disallow: /
bubing disallow: /
cliqzbot disallow: /
cncdialer disallow: /
crawler4j disallow: /
crystalsemanticsbot disallow: /
cyberalert disallow: /
digext disallow: /
discobot disallow: /
discoverybot disallow: /
dloader disallow: /
dloader(naverrobot) disallow: /
doc disallow: /
dotbot disallow: /
download ninja disallow: /
dts agent disallow: /
exabot disallow: /
ezooms disallow: /
fairshare disallow: /
fetch disallow: /
flamingo_searchengine disallow: /
genieo disallow: /
gigabot disallow: /
grub-client disallow: /
heritrix disallow: /
heritrix/3.3.0 disallow: /
httrack disallow: /
integromedb disallow: /
istellabot disallow: /
jikespider disallow: /
jyxobot disallow: /
k2spider disallow: /
kimengi disallow: /
kimengi/nineconnections.com disallow: /
larbin disallow: /
lexxebot/1.0 disallow: /
libwww disallow: /
linko disallow: /
livelapbot disallow: /
magpie-crawler disallow: /
maxthon disallow: /
metauri disallow: /
microsoft.url.control disallow: /
mj12bot disallow: /
moreover disallow: /
moreoverbot disallow: /
msiecrawler disallow: /
nabot disallow: /
naverbot disallow: /
nerdbynature.bot disallow: /
netestate ne crawler disallow: /
netseer crawler disallow: /
newscan disallow: /
nextgensearchbot disallow: /
npbot disallow: /
nutch disallow: /
offline explorer disallow: /
omgilibot disallow: /
orthogaffe disallow: /
piplbot disallow: /
pixray-seeker disallow: /
proximic disallow: /
psbot disallow: /
queryseekerspider disallow: /
rogerbot disallow: /
seokicks disallow: /
seokicks-robot disallow: /
sitebot disallow: /
sitebot/0.1 disallow: /
sitecheck.internetseer.com disallow: /
sitesnagger disallow: /
slurp disallow: /
sogou disallow: /
sosospider disallow: /
spbot disallow: /
spinn3r disallow: /
teleport disallow: /
teleportpro disallow: /
trendictionbot disallow: /
trovitbot disallow: /
turnitinbot disallow: /
ubicrawler disallow: /
umbot-ln disallow: /
unisterbot disallow: /
universalfeedparser disallow: /
wbsearchbot disallow: /
webcopier disallow: /
webreaper disallow: /
webstripper disallow: /
webzip disallow: /
wesee:search disallow: /
wget disallow: /
wotbot disallow: /
wotbox disallow: /
xenu disallow: /
yasni disallow: /
zao disallow: /
zealbot disallow: /
zyborg disallow: /
offlineexplorer disallow: /
chatgpt-user disallow: /
gptbot disallow: /
ccbot disallow: /
anthropic-ai disallow: /
cohere-ai disallow: /
omgili disallow: /
claritybot disallow: /
google-extended disallow: /

No rules defined. All paths allowed.

Comments

• robots.txt para un blog WordPress.
• Bloquear o permitir acceso a contenido adjunto. (Si la instalación está en /public_html).
• Desindexar carpetas que empiecen por wp-
• Permitir Feed general para Google Blogsearch.
• Impedir que /permalink/feed/ sea indexado pues el feed de comentarios suele posicionarse antes de los post.
• Impedir URLs terminadas en /trackback/ que sirven como Trackback URI (contenido duplicado).
• Evita bloqueos de CSS y JS.
• Lista de bots que deberías permitir.
• Lista de bots que generan consultas abusivas aunque siguen las pautas del archivo robots.txt
• Slurp (Yahoo!), Noxtrum y el bot de MSN que suelen generar excesivas consultas.
• lista de bots y ia a bloquear

Warnings

• 1 invalid line.