vaalit.yle.fi
robots.txt

Robots Exclusion Standard data for vaalit.yle.fi

Resource Scan

Scan Details

Site Domain vaalit.yle.fi
Base Domain yle.fi
Scan Status Ok
Last Scan2025-04-02T20:42:57+00:00
Next Scan 2025-05-02T20:42:57+00:00

Last Scan

Scanned2025-04-02T20:42:57+00:00
URL https://vaalit.yle.fi/robots.txt
Domain IPs 13.226.2.113, 13.226.2.4, 13.226.2.51, 13.226.2.92, 2600:9000:248d:0:a:3fa8:340:93a1, 2600:9000:248d:1600:a:3fa8:340:93a1, 2600:9000:248d:2400:a:3fa8:340:93a1, 2600:9000:248d:4000:a:3fa8:340:93a1, 2600:9000:248d:400:a:3fa8:340:93a1, 2600:9000:248d:d000:a:3fa8:340:93a1, 2600:9000:248d:d600:a:3fa8:340:93a1, 2600:9000:248d:fa00:a:3fa8:340:93a1
Response IP 3.164.68.119
Found Yes
Hash 6900833822988e54b16c8336be35614aadfa308e36c7b2de8b90434340eb913e
SimHash 67824bd1e057

Groups

*

Rule Path
Disallow /content/

ai2bot

Rule Path
Disallow /

ai2bot-dolma

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

crawlspace

Rule Path
Disallow /

deepseek

Rule Path
Disallow /

deepseekbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

grok

Rule Path
Disallow /

grokai

Rule Path
Disallow /

iaskspider/2.0

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

isscyberriskcrawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

sidetrade indexer bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

x.ai

Rule Path
Disallow /

xai

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Comments

  • ai crawling