asker.kommune.no
robots.txt

Robots Exclusion Standard data for asker.kommune.no

Resource Scan

Scan Details

Site Domain asker.kommune.no
Base Domain asker.kommune.no
Scan Status Ok
Last Scan2025-07-28T15:35:10+00:00
Next Scan 2025-08-27T15:35:10+00:00

Last Scan

Scanned2025-07-28T15:35:10+00:00
URL https://asker.kommune.no/robots.txt
Redirect https://www.asker.kommune.no/robots.txt
Redirect Domain www.asker.kommune.no
Redirect Base asker.kommune.no
Domain IPs 194.63.248.52, 2a01:5b40:0:248::52
Redirect IPs 104.18.35.5, 172.64.152.251, 2606:4700:4400::6812:2305, 2606:4700:4400::ac40:98fb
Response IP 104.18.35.5
Found Yes
Hash c8caf5a36a83e5f920dc3f0e7061f59181e036c49f5c496872ee431a661a6a16
SimHash 630ac9d1c5e5

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

slurp

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

googleother

Rule Path
Allow /

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

ai2bot

Rule Path
Disallow /

ai2bot-dolma

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleother-image

Rule Path
Disallow /

googleother-video

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

iaskspider

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

isscyberriskcrawler

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

sidetrade

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.asker.kommune.no/sitemap.xml

Comments

  • Allow legitimate search engines
  • Explicit allow some ai bots
  • Default rule
  • Block AI training crawlers