Log In Sign Up

nlt.se
robots.txt

Robots Exclusion Standard data for nlt.se

Resource Scan

Scan Details

Site Domain nlt.se
Base Domain nlt.se
Scan Status Ok
Last Scan2025-10-08T17:00:48+00:00
Next Scan 2025-10-15T17:00:48+00:00

Last Scan

Scanned2025-10-08T17:00:48+00:00
URL https://nlt.se/robots.txt
Redirect https://www.nlt.se:443/robots.txt
Redirect Domain www.nlt.se
Redirect Base nlt.se
Domain IPs 15.197.214.81
Redirect IPs 104.26.12.38, 104.26.13.38, 172.67.70.74, 2606:4700:20::681a:c26, 2606:4700:20::681a:d26, 2606:4700:20::ac43:464a
Response IP 172.67.70.74
Found Yes
Hash fdff2eeb0fff91d2dbefa7610175dfb1cf2f983b3a5b32eb6687032ea3e8fc17
SimHash 651483d88fb0

Groups

adsbot-google
*

Rule Path
Disallow /min-sida/
Disallow /sok/

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

giant oak mn

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.nlt.se/sitemaps/news/sitemap.xml
sitemap https://www.nlt.se/sitemaps/static/sitemap.xml
Back to top

Comments

  • AdsBot crawlers must be named explicitly
  • Sitemaps
  • Disallow Rules
  • Awario (https://awario.com/)
  • Disallow AI crawlers
  • Common crawl
  • OpenAI (ChatGPT)
  • OpenAI (ChatGPT realtime search)
  • OpenAI search bot
  • Anthropic
  • Google Bard, Vertex AI, Gemini
  • webz.io robot
  • webz.io robot
  • FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology.
Back to top