Log In Sign Up

dagbladet.no
robots.txt

Robots Exclusion Standard data for dagbladet.no

Resource Scan

Scan Details

Site Domain dagbladet.no
Base Domain dagbladet.no
Scan Status Ok
Last Scan2024-11-09T20:34:44+00:00
Next Scan 2024-11-16T20:34:44+00:00

Last Scan

Scanned2024-11-09T20:34:44+00:00
URL https://dagbladet.no/robots.txt
Redirect https://www.dagbladet.no/robots.txt
Redirect Domain www.dagbladet.no
Redirect Base dagbladet.no
Domain IPs 104.18.34.161, 172.64.153.95, 2606:4700:4400::6812:22a1, 2606:4700:4400::ac40:995f
Redirect IPs 104.18.34.161, 172.64.153.95, 2606:4700:4400::6812:22a1, 2606:4700:4400::ac40:995f
Response IP 172.64.153.95
Found Yes
Hash 570c6ede9f043a10e757fc4a1fa72cecc1b8bafffe98a1eb7d7d0fcc5388cecc
SimHash 570765e1c477

Groups

*

Rule Path
Disallow *.cgi
Disallow /2009/06/18/nyheter/iran/valg/innenriks/politikk/6784480/
Disallow /2009/09/21/nyheter/innenriks/giftdrap/8225100/
Disallow /2011/09/01/tema/barn/familie/oppvekst/foreldre/17916046/
Disallow /2011/12/05/kultur/debatt/debattinnlegg/anders_behring_breivik/ideologi/19288301/
Disallow /2013/07/12/nyheter/innenriks/giftmorder/terje_wiik/28160672/
Disallow /2013/09/29/nyheter/utenriks/terror/kenya/westgate/29515561/
Disallow /tag/saera_khan
Disallow /tag/terje_wiik
Allow /eavis/_next/
Disallow /eavis/
Disallow /youtube-player/
Disallow /video/playlist/
Disallow /video/popular/
Disallow /video/
Disallow /emne/terje%20erlid
Allow /app/sanity-cdn/images/
Allow /app/sanity-next-app/
Allow /app/raven-frontend/
Allow /app/showcasr/
Allow /app/cyclops-router-api/api/v1/
Allow /app/jw-api-proxy/sitemap/index/dagbladet.xml
Disallow /app/
Allow /video/*/*

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.dagbladet.no/sitemapindex.xml
sitemap https://www.dagbladet.no/app/jw-api-proxy/sitemap/index/dagbladet.xml
Back to top

Comments

  • Questions: webmaster@dagbladet.no
Back to top