Log In Sign Up

eb.dk
robots.txt

Robots Exclusion Standard data for eb.dk

Resource Scan

Scan Details

Site Domain eb.dk
Base Domain eb.dk
Scan Status Ok
Last Scan2024-11-15T19:25:43+00:00
Next Scan 2024-11-22T19:25:43+00:00

Last Scan

Scanned2024-11-15T19:25:43+00:00
URL https://eb.dk/robots.txt
Redirect https://ekstrabladet.dk/robots.txt
Redirect Domain ekstrabladet.dk
Redirect Base ekstrabladet.dk
Domain IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 151.101.1.91
Found Yes
Hash dca0e31a9c5cfb715704eb7b3a74474cd27067dc158c7d7ad5359f86a06a2577
SimHash 52179450e514

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /ritzau_new/
Disallow /ebtv/
Disallow /component/
Disallow /payflow/
Disallow /bibliotek/
Disallow /afstemninger/
Disallow /backoffice/
Disallow /EbNewsConfiguration/
Disallow /side9/
Back to top

Other Records

Field Value
sitemap https://ekstrabladet.dk/svc/sitemap/
sitemap https://ekstrabladet.dk/advertorials/sitemap.xml
Back to top

Comments

  • robots.txt, ekstrabladet.dk
  • AI crawler reference
  • The link below provides instructions to what kind of content can be used to train AI models on this website
  • https://ekstrabladet.dk/ai.txt
  • Common crawl
  • OpenAI (ChatGPT)
  • OpenAI (ChatGPT realtime search)
  • Anthropic
  • Perplexity
  • adsense
Back to top