Log In Sign Up

www.media.volvocars.com
robots.txt

Robots Exclusion Standard data for www.media.volvocars.com

Resource Scan

Scan Details

Site Domain www.media.volvocars.com
Base Domain volvocars.com
Scan Status Ok
Last Scan2024-09-02T21:56:29+00:00
Next Scan 2024-10-02T21:56:29+00:00

Last Scan

Scanned2024-09-02T21:56:29+00:00
URL https://www.media.volvocars.com/robots.txt
Domain IPs 15.156.154.110, 3.98.252.51, 35.183.247.62
Response IP 3.98.252.51
Found Yes
Hash 049424cfdb3df2ecde5337bf3b98d44f500805b3fd030155c115e9d8c36a4346
SimHash 72339949c330

Groups

ahrefsbot
ezooms
sistrix
mj12bot
megaindex.ru
megaindex.com
petalbot

Rule Path
Disallow /

ccbot
claudebot
claude-web
chatgpt-user
gptbot
google-extended
applebot-extended
anthropic-ai
omgilibot
omgili
facebookbot
diffbot
bytespider
imagesiftbot
perplexitybot
cohere-ai

Rule Path
Disallow /

*

Rule Path
Disallow /*/*/search/
Disallow /*/*/print/
Disallow /*/*/download/*
Disallow /*/*/basket/*
Disallow /*/enhanced/
Disallow /Content/Compiled/
Disallow /Content/JQueryUI/
Disallow /Content/Flash/
Disallow /Content/assets/
Disallow /Scripts/
Disallow /Style/
Disallow /ru/

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://www.media.volvocars.com/sitemap_index.xml
Back to top

Comments

  • AI Data Scrapers
  • ----------------
  • This list of bots based on https://darkvisitors.com/ and https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
  • Info on the different bots is possible at https://darkvisitors.com/
Back to top