Log In Sign Up

teknikhistoria.nyteknik.se
robots.txt

Robots Exclusion Standard data for teknikhistoria.nyteknik.se

Resource Scan

Scan Details

Site Domain teknikhistoria.nyteknik.se
Base Domain nyteknik.se
Scan Status Ok
Last Scan2025-10-21T20:30:13+00:00
Next Scan 2025-11-20T20:30:13+00:00

Last Scan

Scanned2025-10-21T20:30:13+00:00
URL https://teknikhistoria.nyteknik.se/robots.txt
Domain IPs 2600:9000:28c2:3000:17:e17d:4fc0:93a1, 2600:9000:28c2:3e00:17:e17d:4fc0:93a1, 2600:9000:28c2:6e00:17:e17d:4fc0:93a1, 2600:9000:28c2:7a00:17:e17d:4fc0:93a1, 2600:9000:28c2:9a00:17:e17d:4fc0:93a1, 2600:9000:28c2:b400:17:e17d:4fc0:93a1, 2600:9000:28c2:be00:17:e17d:4fc0:93a1, 2600:9000:28c2:d800:17:e17d:4fc0:93a1, 3.171.198.28, 3.171.198.55, 3.171.198.67, 3.171.198.93
Response IP 3.171.198.28
Found Yes
Hash 3fab68926e09a7f9c45078994a1d3aa3df26ab9e5671796677fbc26de8820284
SimHash 50f04942e7fe

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gemini

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /
Back to top

Comments

  • Block AI-related bots used for model training or LLM-based features
Back to top