nordatanet.no
robots.txt
Robots Exclusion Standard data for nordatanet.no
Resource Scan
Scan Details
| Site Domain | nordatanet.no |
| Base Domain | nordatanet.no |
| Scan Status | Ok |
| Last Scan | 2025-08-18T00:11:02+00:00 |
| Next Scan | 2025-09-17T00:11:02+00:00 |
Last Scan
| Scanned | 2025-08-18T00:11:02+00:00 |
| URL | https://nordatanet.no/robots.txt |
| Redirect | https://www.nordatanet.no/robots.txt |
| Redirect Domain | www.nordatanet.no |
| Redirect Base | nordatanet.no |
| Domain IPs | 157.249.72.82 |
| Redirect IPs | 157.249.72.82 |
| Response IP | 157.249.72.82 |
| Found | Yes |
| Hash | 1d3de13735ec5feebc374a15a172ee966a279aa4d4ef0485d29d53f206c78186 |
| SimHash | 7d06675006b5 |
Groups
amazonbot
anthropic-ai
applebot
applebot-extended
barkrowler
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dalvik/2.1.0
dataprovider
diffbot
cotbot
expanse
censysinspect
facebookbot
friendlycrawler
go-http-client
google-extended
googleother
googleother-image
googleother-video
gptbot
http.rb*
imagesiftbot
img2dataset
internet-measurement
meta-externalfetcher
meta-externalagent
meltwater
mj12bot
news-please
oai-searchbot
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
petalbot
timipibot
youbot
facebookexternalhit
twitterbot
zoominfobot
adbeat_bot
adsbot-google
adsbot-google-mobile
aihitbot
anderspinkbot
archivebot
awariobot
awariosmartbot
bitsightbot
blackboard
brandverity
cincraw
ev-crawler
google-safety
hubspot
ioncrawl
jugendschutzprogramm-crawler
kstandbot
lightspeedsystemscrawler
linkfluence
linkwalker
magpie-crawler
mediapartners-google
mediatoolkitbot
muckrack
netcraftsurveyagent
netvibes
pandalytics
panscient.com
proximic
scoop.it
seekportbot
smtbot
trendictionbot
trendsmapresolver
turnitin
turnitinbot
tweetmemebot
twingly
um-ln
velenpublicwebcrawler
virustotal
zoominfobot
dataprovider.com
dcrawl
httrack
httrack 3.0
metainspector
newspaper
nutch
offline explorer
openindexspider
ahrefsbot
barkrowler
blexbot
brightedge crawler
cocolyzebot
dataforseobot
domainstatsbot
dotbot
hypestat
linkdexbot
mj12bot
online-webceo-bot
screaming frog seo spider
semrushbot
semrushbot-ba
semrushbot-ct
semrushbot-si
semrushbot-swa
senutobot
seobilitybot
seokicks
seolizer
serpstatbot
sitecheckerbotcrawler
zoombot
a patent crawler
a6-indexer
aboundex
academicbotrtu
acapbot
acoonbot
acunetix security scanner
acunetix web vulnerability scanner
addsearchbot
addthis
adequat
adequat-systems
adidxbot
admantx
adscanner
adstxtcrawler
advbot
aisearchbot
alexabot
alexibot
alphabot
amisoftware
antibot
anyevent
apercite
appinsights
aqua_products
arabot
ask n read
asknread.com
aspiegelbot
asterias
augure
auramundi
awariorssbot
awesomecrawler
b2b bot
b2w
backdoorbot
backlinkcrawler
baidu-yunguance
baiduspider-image
baiduspider-news
baiduspider-video
bdcbot
behloolbot
betabot
better uptime bot
bidswitchbot
biglotron
binlar
birdcrawlerbot
bitbot
black hole
blekkobot
blogmurabot
blowfish
blp_bbot
bnf.fr_bot
bomborabot
bookmark search tool
bot-pge.chlooe.com
bot.araturka.com
botalot
botify
botrighthere
boxcarbot
brainobot
brandonbot
btwebclient
bubing
buck
builtbottough
bullseye
bunnyslippers
buzzbot
caliperbot
capsulechecker
careerbot
cc metadata scaper
cegbfeieh
centurybot
changedetection
checkmarknetwork
cheesebot
cherrypicker
cherrypickerelite
cherrypickerse
cision
cispa webcrawler
citeseerxbot
citoid
claritybot
clickagy
cliqzbot
cloudflare-alwaysonline
coccoc
coccocbot
coexel
companybook-crawler
content crawler spider
contextad bot
contxbot
convera
converacrawler
cookiebot
copernic
copyrightcheck
corporama
cosmos
crawler4j
crawlyprojectcrawler
crescent
crescent internet toolpak http ole control v.1.0
criteobot
crunchbot
crystalsemanticsbot
curebot
custom-asynchttpclient
cutbot
cxensebot
cyberpatrol
dareboost
datafeedwatch
datagnionbot
datanyze
daumoa
deepcrawl
deepnoc
deusu
digg deeper
digimind
digincore bot
discobot
disqus
dittospyder
dnyzbot
domain re-animator bot
domaincrawler
dow jones searchbot
download ninja
dragonbot
drupact
dubbotbot
e.ventures investment crawler
easybib autocite
ec2linkfinder
edisterbot
electricmonk
elisabot
ellisphere
emailcollector
emailsiphon
emailwolf
epicbot
eright
erocrawler
etaospider
europarchive.org
evc-batch
everyonesocialbot
exabot
experibot
extlinksbot
extractorpro
eyeotabot
ezid
ezooms
facebot
fairad client
fast enterprise crawler
fast-webcrawler
fedidb
fedoraplanet
feedbin
feedbot
feedburner
feedspot
feedvalidator
femtosearchbot
fever
finditanswersbot
findlink
findthatfile
findxbot
flaming attackbot
flamingo_searchengine
fluffy
foobot
fr-crawler
freewebmonitoring sitechecker
freshpingbot
fuelbot
fyrebot
g00g1e.net
g2 web services
g2reader-bot
gaisbot
garlikcrawler
genieo
getright
gigablast
gigabot
gingercrawler
gluten free crawler
gnam gnam spider
gnowitnewsbot
google-adwords-instant
google-certificates-bridge
google-physicalweb
google-site-verification
google-structured-data-testing-tool
google-xrawler
gowikibot
grapeshot
grapeshotcrawler
grobbot
grouphigh
grub-client
grub.org
gsa-crawler
gslfbot
gwene
harvest
hawaiibot
humanlinks
hyscore.io
ias crawler
icbot
icc-crawler
ichiro
imrbot
indeedbot
inetdex-bot
infonavirobot
infoobot
infoseek
integromedb
intelium_bot
interfaxscanbot
ip-web-crawler.com
irlbot
iron33
iskanie
israbot
istellabot
it2media-domain-crawler
james bot
jamesbot
jamie's spider
jenkersbot
jennybot
jetbot
jetty
jikespider
jobboersebot
jooblebot
jpg-newsbot
jyxobot
k2spider
k7mlwcbot
kbcrawl
kemvibot
kenjin spider
keys-so-bot
keyword density
knowings
komodiabot
kosmiobot
landau-media-spider
larbin
laserlikebot
lb-spider
leadbox
leikibot
lexibot
libweb
linespider
linguee bot
linkapediabot
linkarchiver
linkcheck by siteimprove.com
linkdex
linkextractorpro
linkisbot
linko
linkpadbot
linkscan
lipperhey
livelapbot
lkxscan
lnspiderguy
lssbot
lssrocketcrawler
ltx71
luminator-robots
lwp-trivial
macocu
mappydata
mata hari
mauibot
mbcrawler
megaindex
megaindex.ru
meltawer
meltwater
meltwaternews
memorybot
mention
metajobbot
metauri
miixpc
mindupbot
minicrawler
mister pix
mixnodecache
mlbot
moatbot
moget
mojeek
moodlebot
moreover
ms search 4.0 robot
ms search 6.0 robot
msiecrawler
msrbot
mtrobot
multiviewbot
mytwip
naver blog rssbot
naverbot
neevabot
nerdbynature.bot
nerdybot
netants
netestate ne crawler
neticle crawler
netmechanic
netresearchserver
netsystemsresearch
newsharecounts
newsnow
newzbin
nextgensearchbot
nicerspro
niki-bot
nimblecrawler
nimbostratus-bot
ninja bot
nixstatsbot
nlux_iaharvester
nmap scripting engine
npbot
ntentbot
nuzzel
odklbot
officestorebot
omgilibot
openbot
openfind
openfind data gatherer
opengraphcheck
openhosebot
opinion-tracker
oracle ultra search
orangebot
orthogaffe
outbrain
outclicksbot
page2rss
pagepeeker
pagething
perman
pingdom
pinterest
piplbot
postrank
pr-cy.ru
primalbot
privacyawarebot
propowerbot
prowebwalker
proxem
psbot
pulsepoint
purebot
queryn metasearch
qwam content intelligence
radiation retriever 1.1
rankactivelinkbot
rankflex
refindbot
regionstuttgartbot
repomonkey
repomonkey bait & tackle
retrevopageanalyzer
reverseengineeringbot
ridderbot
riddler
rivva
robozilla
rssbot
rssingbot
rukicrawler
ruxitsynthetic
rytebot
safednsbot
safesearch microdata crawler
sbl-bot
score3
scoutjet
scribdbot
scrubby
search.marginalia.nu
searchatlas
searchmetricsbot
searchpreview
seekbot
seekport crawler
seekr
seewithkids
semanticbot
sempi.tech
semrushbot-bm
semrushbot-sa
sentibot
seokicks-robot
seoscanners
seostar.co
seostats
simplecrawler
simplescraper
sindup
sistrix crawler
sitebot
sitecheck.internetseer.com
siteexplorer.info
siteimprove
siteimprove.com
sitesnagger
sitesucker
slack-imgproxy
slackbot
slurp
socialrankiobot
sogou
sogou inst spider
sogou spider2
sonic
sosospider
spankbot
spanner
spbot
spinn3r
spotter
sputnikbot
storebot-google
storygizebot
stractbot
streamline3bot
summalybot
summify
superbot
surveybot
suzuran
swiftbot
swimgbot
synthesio
sysomos
szukacz
taboolabot
tagoobot
talkwater
tangibleebot
teleport
teleportpro
telesoft
the intraformant
thenomad
theoldreader.com
thinklab
tigerbot
titan
tocrawl
tombapublicwebcrawler
toplistbot
toutiaospider
traackr.com
tracemyfile
trafilatura
trendeo
trendkite-akashic-crawler
trendybuzz
trovitbot
true_robot
truliabot
turingos
tweetedtimes
twengabot
twurly
ubicrawler
um-ic
updownerbot
upflow
uptime-kuma
uptimebot.org
uptimerobot
url control
url_spider_pro
urlappendbot
urly warning
usasearch
usinenouvellecrawler
ut-dorkbot
validator.nu
vci
vci webviewer vci webviewer win32
vebidoobot
vecteurplus
veoozbot
verticalsearch
vigil
vkrobot
voilabot
voltron
voluumdsp-content-bot
vsw
vuhuvbot
w3c_i18n-checker
w3c_unicorn
w3c-checklink
w3c-mobileok
wasalive-bot
wbsearchbot
web image collector
web-archive-net.com.bot
webauto
webbandit
webcapture 2.0
webcompanycrawler
webcopier
webcopier v.2.2
webcopier v3.2a
webdatastats
webenhancer
webmasterworldforumbot
webmon
webreaper
websauger
website quester
webstripper
webzip
winello
winhttrack
wiseguys robot
wocbot
woobot
woorankreview
wordupinfosearch
woriobot
wotbox
www-collector-e
www-mechanize
www.uptime.com
xenu
xenu link sleuth
xenu's
xenu's link sleuth 1.1c
xovibot
yahoo pipes 1.0
yak
yandexmobilebot
yandexvideo
yanga
yellowbrandprotectionbot
yoozbot
youdaobot
youmag
zabbix
zao
zealbot
zenback bot
zeus
zeus link scout
zgrab
zite
zuperlistbot
zyborg
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /aen/search |
| Allow | /core/*.css$ |
| Allow | /core/*.css? |
| Allow | /core/*.js$ |
| Allow | /core/*.js? |
| Allow | /core/*.gif |
| Allow | /core/*.jpg |
| Allow | /core/*.jpeg |
| Allow | /core/*.png |
| Allow | /core/*.svg |
| Allow | /profiles/*.css$ |
| Allow | /profiles/*.css? |
| Allow | /profiles/*.js$ |
| Allow | /profiles/*.js? |
| Allow | /profiles/*.gif |
| Allow | /profiles/*.jpg |
| Allow | /profiles/*.jpeg |
| Allow | /profiles/*.png |
| Allow | /profiles/*.svg |
| Disallow | /core/ |
| Disallow | /profiles/ |
| Disallow | /README.txt |
| Disallow | /web.config |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /user/register |
| Disallow | /user/password |
| Disallow | /user/login |
| Disallow | /user/logout |
| Disallow | /media/oembed |
| Disallow | /*/media/oembed |
| Disallow | /index.php/admin/ |
| Disallow | /index.php/comment/reply/ |
| Disallow | /index.php/filter/tips |
| Disallow | /index.php/node/add/ |
| Disallow | /index.php/search/ |
| Disallow | /index.php/user/password |
| Disallow | /index.php/user/register |
| Disallow | /index.php/user/login |
| Disallow | /index.php/user/logout |
| Disallow | /index.php/media/oembed |
| Disallow | /index.php/*/media/oembed |
| Disallow | /metsis/map/ |
| Disallow | /index.php/metsis/map/ |
| Disallow | /metsis/export/ |
| Disallow | /index.php/metsis/export/ |
| Disallow | /metsis/basket/ |
| Disallow | /index.php/metsis/basket/ |
| Disallow | /metsis/tsplot/ |
| Disallow | /index.php/metsis/tsplot/ |
| Disallow | /metsis/bokeh/ |
| Disallow | /index.php/metsis/bokeh/ |
| Disallow | /metsis/thumbnail/ |
| Disallow | /index.php/metsis/thumbnail/ |
| Disallow | /metsis/thumbnail/ |
| Disallow | /index.php/metsis/thumbnail/ |
| Disallow | /metsis/search |
| Disallow | /index.php/metsis/search |
| Disallow | /metsis/simple_search |
| Disallow | /index.php/metsis/simple_search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Warnings
- 7 invalid lines.
Comments