Log In Sign Up

sla.se
robots.txt

Robots Exclusion Standard data for sla.se

Resource Scan

Scan Details

Site Domain sla.se
Base Domain sla.se
Scan Status Ok
Last Scan2026-02-02T05:54:25+00:00
Next Scan 2026-02-09T05:54:25+00:00

Last Scan

Scanned2026-02-02T05:54:25+00:00
URL https://sla.se/robots.txt
Redirect https://www.sla.se:443/robots.txt
Redirect Domain www.sla.se
Redirect Base sla.se
Domain IPs 15.197.214.81
Redirect IPs 104.26.4.7, 104.26.5.7, 172.67.71.57, 2606:4700:20::681a:407, 2606:4700:20::681a:507, 2606:4700:20::ac43:4739
Response IP 172.67.71.57
Found Yes
Hash 32b6c114629fad7fb83518e6b2c535dcdae35a16936e06401ec789b366f61f53
SimHash 651483d88fb0

Groups

adsbot-google
*

Rule Path
Disallow /min-sida/
Disallow /sok/
Disallow /_migration/

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

giant oak mn

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.sla.se/sitemaps/news/sitemap.xml
sitemap https://www.sla.se/sitemaps/static/sitemap.xml
Back to top

Comments

  • AdsBot crawlers must be named explicitly
  • Sitemaps
  • Disallow Rules
  • Awario (https://awario.com/)
  • Disallow AI crawlers
  • Common crawl
  • OpenAI (ChatGPT)
  • OpenAI (ChatGPT realtime search)
  • OpenAI search bot
  • Anthropic
  • Google Bard, Vertex AI, Gemini
  • webz.io robot
  • webz.io robot
  • FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology.
Back to top