Log In Sign Up

vg.no
robots.txt

Robots Exclusion Standard data for vg.no

Resource Scan

Scan Details

Site Domain vg.no
Base Domain vg.no
Scan Status Ok
Last Scan2024-10-02T10:25:35+00:00
Next Scan 2024-10-09T10:25:35+00:00

Last Scan

Scanned2024-10-02T10:25:35+00:00
URL https://vg.no/robots.txt
Redirect https://www.vg.no/robots.txt
Redirect Domain www.vg.no
Redirect Base vg.no
Domain IPs 195.88.54.16, 195.88.55.16, 2001:67c:21e0::16
Redirect IPs 195.88.54.16, 195.88.55.16, 2001:67c:21e0::16
Response IP 195.88.55.16
Found Yes
Hash a28cccef57a3d1fd72e62da62943792d00c280f59d5fac0faef805323e425eee
SimHash c01d6952d4b7

Groups

*

Rule Path
Disallow /tegneserier/salesposter
Disallow /poll
Disallow /informasjon/redaksjonelle-avgjorelser/182
Disallow /informasjon/redaksjonelle-avgjorelser/212
Disallow /sok*?*
Disallow /?vcheck=1
Disallow /?_escaped_fragment_=%2Fkategori%2F144%2Fbolig-tv
Disallow /?_escaped_fragment_=%2Fkategori

googlebot-news

Rule Path
Disallow /annonsorinnhold/
Disallow /kommersielt-innhold/
Disallow /innstikk/
Disallow /?embed=true
Disallow /arkiv/
Disallow /front/
Disallow /ads/prewarm/
Disallow /adblock-survey/

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

omgili

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.vg.no/sitemaps/files/articles-48hrs.xml
sitemap https://www.vg.no/sitemap/files/articles-48hrs.xml
sitemap https://www.vg.no/sitemap.xml
Back to top

Comments

  • Start AI crawler block
  • VG does not permit unlicensed use of our content for training large language models or other
  • artificial intelligence technology. All use, reproductions and extractions of our content for
  • such purposes require specific approval by VG. We explicitly disallow text and data mining and
  • all other technical means designed to or with the effect that they enable unlicensed use,
  • reproduction or extraction of our content for training AI models.
  • End AI crawler block
Back to top