Log In Sign Up

borssnack.di.se
robots.txt

Robots Exclusion Standard data for borssnack.di.se

Resource Scan

Scan Details

Site Domain borssnack.di.se
Base Domain di.se
Scan Status Ok
Last Scan2025-12-05T11:52:15+00:00
Next Scan 2025-12-12T11:52:15+00:00

Last Scan

Scanned2025-12-05T11:52:15+00:00
URL https://borssnack.di.se/robots.txt
Redirect https://www.di.se/robots.txt
Redirect Domain www.di.se
Redirect Base di.se
Domain IPs 34.117.105.189
Redirect IPs 146.75.117.91, 2a04:4e42:9::347
Response IP 151.101.37.91
Found Yes
Hash f3f6572a1556f22d610620a27c235557912177e8ca92ed45a34dcc37395f1ef2
SimHash 7a2991508d66

Groups

*

Rule Path
Disallow /_alive/
Disallow /_alive
Disallow /login*
Disallow /search*
Disallow /bors/search*
Disallow /*?tab=
Disallow /konto/*
Disallow /insider-bolag/
Disallow /finansiell-information/pressreleaser-per-foretag/
Disallow /finansiell-information/pressreleaser/?page=
Disallow /insider-person/
Disallow /Comments/WebServices/
Disallow /*?fhtab=
Disallow /*%26fhtab%3D
Disallow /*?allakommentarer=
Disallow /*%26allakommentarer%3D
Disallow /*?timestamp=
Disallow /*%26timestamp%3D
Disallow /*?t=
Disallow /*%26t%3D
Disallow /*?flik=
Disallow /*%26flik%3D
Disallow /*?qr=
Disallow /*%26qr%3D
Disallow /*?screenwidth
Disallow /*%26screenwidth
Disallow /*?screenheight
Disallow /*%26screenheight
Disallow /*?ctl
Disallow /*%26ctl
Disallow /*?currentIndex=
Disallow /*?ns_
Disallow /*%26ns_
Disallow /bip/
Disallow /bip-callback*
Disallow /register*
Disallow /part*
Disallow /akademi/anmalan/*
Disallow /akademi/info-utbildningar/
Disallow /34405621/bn/*
Disallow /di-fonder/*
Disallow /_akademi/
Disallow /_akademi
Disallow /logout
Disallow /walls/freemium/*
Disallow /walls/premium/*

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.di.se/sitemap.xml
sitemap https://www.di.se/sitemap-news.xml
Back to top

Comments

  • Common Crawl robot, the resulting dataset is the primary training corpus in every LLM.
  • ChatGPT robot, used to improve the ChatGPT LLM.
  • ChatGPT robot, may be used to improve the ChatGPT LLM.
  • Robot used to improve Bard and Vertex AI LLMs.
  • webz.io robot, the resulting dataset can and is purchased to train LLMs.
  • webz.io robot, the resulting dataset can and is purchased to train LLMs.
  • FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology.
  • Robot used to improve Anthropic AI LLMs.
  • OpenAI search bot
Back to top