eldiariodecoahuila.com
robots.txt

Robots Exclusion Standard data for eldiariodecoahuila.com

Resource Scan

Scan Details

Site Domain eldiariodecoahuila.com
Base Domain eldiariodecoahuila.com
Scan Status Ok
Last Scan2024-06-16T04:20:37+00:00
Next Scan 2024-06-23T04:20:37+00:00

Last Scan

Scanned2024-06-16T04:20:37+00:00
URL http://eldiariodecoahuila.com/robots.txt
Domain IPs 68.66.232.174
Response IP 68.66.232.174
Found Yes
Hash b42cc947768a083e8a51a6986d579e56221629b79d8e09b32a5cd96df4e217d0
SimHash 6a2452513c30

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /respaldo/
Disallow /u/

Other Records

Field Value
crawl-delay 60

facebookbot

Rule Path
Disallow /wp-includes/

Other Records

Field Value Comment
crawl-delay 15 1 page per 15 seconds

facebookexternalhit

Rule Path
Disallow /wp-includes/

Other Records

Field Value Comment
crawl-delay 15 1 page per 15 seconds

*

Rule Path
Disallow /favicon.ico

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 15

turnitinbot

Rule Path
Disallow /

okhttp/2.5.0

Rule Path
Disallow /

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /site.webmanifest

mediapartners-google

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

grapeshot

Rule Path
Disallow

amazonbot

Rule Path
Disallow /do-not-crawl/

sentibot

Rule Path
Disallow /

bloomberg-net

Rule Path
Disallow /wp-includes/

Other Records

Field Value Comment
crawl-delay 10 1 page per 5 seconds

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

yeti

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

buck

Rule Path
Disallow /

wikido

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

g-i-g-a-b-o-t

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

caam

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

jersey

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

omgili

Rule Path
Disallow /

piplbot

Rule Path
Disallow /