cortezjournal.com
robots.txt
Robots Exclusion Standard data for cortezjournal.com
Resource Scan
Scan Details
| Site Domain | cortezjournal.com |
| Base Domain | cortezjournal.com |
| Scan Status | Ok |
| Last Scan | 2024-06-12T07:24:48+00:00 |
| Next Scan | 2024-06-19T07:24:48+00:00 |
Last Scan
| Scanned | 2024-06-12T07:24:48+00:00 |
| URL | http://cortezjournal.com/robots.txt |
| Redirect | https://nsr.the-journal.com/robots.txt |
| Redirect Domain | nsr.the-journal.com |
| Redirect Base | the-journal.com |
| Domain IPs | 52.10.141.130 |
| Redirect IPs | 104.21.55.85, 172.67.146.90, 2606:4700:3034::ac43:925a, 2606:4700:3035::6815:3755 |
| Response IP | 104.21.55.85 |
| Found | Yes |
| Hash | 07e26bb9027780a02917e7cfde581d0332b47723fbf54e0144041d039865055a |
| SimHash | 437848104e90 |
Groups
a6-indexer
ahrefsbot
aliveadvisorcrawler
alphaseobot
alphaseobot-sa
anonymous coward
apollobot
baiduspider
baiduspider-image
baiduspider-video
barkrowler
bitvorebot
blekkobot
blexbot
brandverity/1.0
btcrawler
bubing
buck
buck/2.2
caam
clarsentiabot
clinecrawler
cliqzbot
companybook-crawler
dataprovider.com
domaincrawler
elefent
exabot
exabot-thumbnails
expo9
ezooms
fairshare.cc
fast enterprise crawler
flamingo_searchengine
flipboard
flipboardproxy
fr_crawler
garlikcrawler
genieo
gigabot
g-i-g-a-b-o-t
gnowitnewsbot
goodzer
grapeshot
heritrix
integromedb
laserlikebot
linguee bot
ltx71
lumtelbot
magpie-crawler
mail.ru_bot
mandalay
mauibot
maxpointcrawler
mediawords
meltwaternews
memonewsbot
mj12bot
mojeekbot
moreoverbot
netestate ne crawler
newslookup-bot
newsnow
panscient.com
paperlibot
psbot
piplbot
proximic
rssingbot
qwantify
qwant-news/2.0
r6_commentreader
r6_feedfetcher
rediffnewsbot
riddler
rogerbot
scalaj-http
scalaj-http/1.0
scrapy
semrushbot
seokicks-robot
shakoo
smartbriefbot
sogou spider
sosospider
spbot
spinn3r
superfeedr
superfeedr bot
synthesio
tencenttraveler
test bot
the knowledge ai
toscrawler
toutiaospider
trendictionbot
trendkite-akashic-crawler
turnitinbot
uipbot
vegi bot
veooz
veooz/1.0
vocusbot
wikido
wotbox
yandex
yandexbot
yandeximages
yandexnews
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /admin/ |
| Disallow | /saxotech_importer/ |
| Disallow | /api/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://nsr.the-journal.com/sitemaps/sitemaps.xml.gz |
Warnings
- 1 invalid line.