durangoherald.com
robots.txt
Robots Exclusion Standard data for durangoherald.com
Resource Scan
Scan Details
| Site Domain | durangoherald.com |
| Base Domain | durangoherald.com |
| Scan Status | Ok |
| Last Scan | 2024-06-02T05:19:02+00:00 |
| Next Scan | 2024-06-09T05:19:02+00:00 |
Last Scan
| Scanned | 2024-06-02T05:19:02+00:00 |
| URL | https://durangoherald.com/robots.txt |
| Domain IPs | 107.154.109.252, 107.154.240.252 |
| Response IP | 107.154.240.252 |
| Found | Yes |
| Hash | a7d5e6b676e6b03b5e2eea16e642b4fe868ebe4fcc7122b90caa3c2c21d9f168 |
| SimHash | 537848104e90 |
Groups
a6-indexer
ahrefsbot
aliveadvisorcrawler
alphaseobot
alphaseobot-sa
anonymous coward
apollobot
baiduspider
baiduspider-image
baiduspider-video
barkrowler
bitvorebot
blekkobot
blexbot
brandverity/1.0
btcrawler
bubing
buck
buck/2.2
caam
clarsentiabot
clinecrawler
cliqzbot
companybook-crawler
dataprovider.com
domaincrawler
elefent
exabot
exabot-thumbnails
expo9
ezooms
fairshare.cc
fast enterprise crawler
flamingo_searchengine
flipboard
flipboardproxy
fr_crawler
garlikcrawler
genieo
gigabot
g-i-g-a-b-o-t
gnowitnewsbot
goodzer
heritrix
integromedb
laserlikebot
linguee bot
ltx71
lumtelbot
magpie-crawler
mail.ru_bot
mandalay
mauibot
maxpointcrawler
mediawords
meltwaternews
memonewsbot
mj12bot
mojeekbot
moreoverbot
netestate ne crawler
newslookup-bot
newsnow
panscient.com
paperlibot
psbot
piplbot
proximic
rssingbot
qwantify
qwant-news/2.0
r6_commentreader
r6_feedfetcher
rediffnewsbot
riddler
rogerbot
scalaj-http
scalaj-http/1.0
scrapy
semrushbot
seokicks-robot
shakoo
smartbriefbot
sogou spider
sosospider
spbot
spinn3r
superfeedr
superfeedr bot
synthesio
tencenttraveler
test bot
the knowledge ai
toscrawler
toutiaospider
trendictionbot
trendkite-akashic-crawler
turnitinbot
uipbot
vegi bot
veooz
veooz/1.0
vocusbot
wikido
wotbox
yandex
yandexbot
yandeximages
yandexnews
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /admin/ |
| Disallow | /saxotech_importer/ |
| Disallow | /api/ |
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Warnings
- 1 invalid line.