Log In Sign Up

ekstrabladet.dk
robots.txt

Robots Exclusion Standard data for ekstrabladet.dk

Resource Scan

Scan Details

Site Domain ekstrabladet.dk
Base Domain ekstrabladet.dk
Scan Status Ok
Last Scan2024-11-13T17:10:35+00:00
Next Scan 2024-11-20T17:10:35+00:00

Last Scan

Scanned2024-11-13T17:10:35+00:00
URL https://ekstrabladet.dk/robots.txt
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 151.101.129.91
Found Yes
Hash 599b43c2a60a4bbea79bf0d5ced976370a8f4f1ebea90cc58569261841093d69
SimHash 5a179550e524

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /ritzau_new/
Disallow /ebtv/
Disallow /component/
Disallow /payflow/
Disallow /bibliotek/
Disallow /afstemninger/
Disallow /backoffice/
Disallow /EbNewsConfiguration/
Disallow /side9/
Back to top

Other Records

Field Value
sitemap https://ekstrabladet.dk/svc/sitemap/
sitemap https://ekstrabladet.dk/advertorials/sitemap.xml
Back to top

Comments

  • robots.txt, ekstrabladet.dk
  • AI crawler reference
  • The link below provides instructions to what kind of content can be used to train AI models on this website
  • https://ekstrabladet.dk/ai.txt
  • Common crawl
  • OpenAI (ChatGPT)
  • OpenAI (ChatGPT realtime search)
  • Anthropic
  • adsense
Back to top