Log In Sign Up

kt.se
robots.txt

Robots Exclusion Standard data for kt.se

Resource Scan

Scan Details

Site Domain kt.se
Base Domain kt.se
Scan Status Ok
Last Scan2025-11-28T21:22:27+00:00
Next Scan 2025-12-05T21:22:27+00:00

Last Scan

Scanned2025-11-28T21:22:27+00:00
URL https://kt.se/robots.txt
Redirect https://www.kt.se:443/robots.txt
Redirect Domain www.kt.se
Redirect Base kt.se
Domain IPs 15.197.214.81
Redirect IPs 104.21.33.49, 172.67.158.222, 2606:4700:3031::6815:2131, 2606:4700:3033::ac43:9ede
Response IP 104.21.33.49
Found Yes
Hash 1b425640c868034189d0f8a8ec770b516a269c84e7f58e44d66d4bb332e2443b
SimHash 65148358cfb0

Groups

adsbot-google
*

Rule Path
Disallow /min-sida/
Disallow /sok/

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

giant oak mn

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.kt.se/sitemaps/news/sitemap.xml
sitemap https://www.kt.se/sitemaps/static/sitemap.xml
Back to top

Comments

  • AdsBot crawlers must be named explicitly
  • Sitemaps
  • Disallow Rules
  • Awario (https://awario.com/)
  • Disallow AI crawlers
  • Common crawl
  • OpenAI (ChatGPT)
  • OpenAI (ChatGPT realtime search)
  • OpenAI search bot
  • Anthropic
  • Google Bard, Vertex AI, Gemini
  • webz.io robot
  • webz.io robot
  • FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology.
Back to top