elmundo.com
robots.txt

Robots Exclusion Standard data for elmundo.com

Resource Scan

Scan Details

Site Domain elmundo.com
Base Domain elmundo.com
Scan Status Ok
Last Scan2024-11-13T16:33:51+00:00
Next Scan 2024-11-20T16:33:51+00:00

Last Scan

Scanned2024-11-13T16:33:51+00:00
URL https://elmundo.com/robots.txt
Redirect https://www.elmundo.com/robots.txt
Redirect Domain www.elmundo.com
Redirect Base elmundo.com
Domain IPs 131.0.136.66
Redirect IPs 131.0.136.66
Response IP 131.0.136.66
Found Yes
Hash 1815201a0117a80f18999f9102089c437d925dddd8764d4913a6050e52f6760a
SimHash ccfb1b80d5f5

Groups

*

Rule Path
Disallow /assets/
Disallow /recursos/

twitterbot

Rule Path
Allow *

facebot

Rule Path
Allow *

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow /0
Disallow /tag/servicio-nacional-de-aprendizaje-%28sena%29
Disallow /tag/sociedad-antioquena-de-astronomia-%28saa%29
Disallow /futbol/fechas
Disallow /mundo/interna/controlar_medicion_periodista
Disallow /mundo/interna/controlar_medicion_lugar
Disallow /mundo/interna/controlar_ver_mas_columnistas
Disallow /mundo/interna/controlar_medicion_redes
Disallow /mundo/interna/ver_mas_etiqueta
Disallow /mundo/interna/controlar_medicion_tag
Disallow /home/ver_mas
Disallow /mundo/interna/controlar_compartir_correo
Disallow /futbol/fechas