clon.dondiario.com
robots.txt

Robots Exclusion Standard data for clon.dondiario.com

Resource Scan

Scan Details

Site Domain clon.dondiario.com
Base Domain dondiario.com
Scan Status Ok
Last Scan2024-04-26T12:28:50+00:00
Next Scan 2024-05-26T12:28:50+00:00

Last Scan

Scanned2024-04-26T12:28:50+00:00
URL https://clon.dondiario.com/robots.txt
Domain IPs 104.26.4.39, 104.26.5.39, 172.67.72.174, 2606:4700:20::681a:427, 2606:4700:20::681a:527, 2606:4700:20::ac43:48ae
Response IP 104.26.4.39
Found Yes
Hash c6cd45e9905a9ef09b72bb9696f392937367d1bb225061be05789367f0e2edc4
SimHash 684cfc908913

Groups

*

Rule Path
Allow /userfiles/
Disallow /admin/
Disallow /donbalonrosa/
Disallow /555/
Disallow /ustedpregunta/categoria/sexo/
Disallow /adconion_preroll/
Disallow /videos/
Disallow /noticia/admin/
Disallow /imagen/admin/
Disallow /encuesta/admin/
Disallow /video/admin/
Disallow /directo/admin/
Disallow /admin/elementos/
Disallow /admin/publicidad/
Disallow /alert/
Disallow /user/admin/

mediapartners-google

Rule Path
Allow /

googlebot

Rule Path
Allow /
Disallow /es_ES/resultados/noticias/?q=
Allow /feed/$
Disallow /feed
Disallow /comments/feed
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://panel.defensacentral.com/sitemap_index.xml
sitemap https://panel.defensacentral.com/news/index.php