Log In Sign Up

ricardo.ch
robots.txt

Robots Exclusion Standard data for ricardo.ch

Resource Scan

Scan Details

Site Domain ricardo.ch
Base Domain ricardo.ch
Scan Status Ok
Last Scan2024-06-02T20:42:20+00:00
Next Scan 2024-06-09T20:42:20+00:00

Last Scan

Scanned2024-06-02T20:42:20+00:00
URL https://ricardo.ch/robots.txt
Redirect https://www.ricardo.ch/robots.txt
Redirect Domain www.ricardo.ch
Redirect Base ricardo.ch
Domain IPs 104.18.42.46, 172.64.145.210
Redirect IPs 104.18.42.46, 172.64.145.210
Response IP 172.64.145.210
Found Yes
Hash bd51d2b36ff3fdcb559389d07b8918d002c8abc075c8ea5091ccd6cf18e37ccb
SimHash 355ecb7c8ffb

Groups

*

Rule Path
Disallow /en/
Disallow /en$
Disallow /de/login
Disallow /fr/login
Disallow /it/login
Disallow /de/regist
Disallow /fr/regist
Disallow /it/regist
Disallow /*/bulk/landing
Disallow /*/my-ricardo
Disallow /*/bookkeeping
Disallow /de/profile
Disallow /fr/profile
Disallow /it/profile
Disallow /de/list
Disallow /fr/list
Disallow /it/list
Disallow /*/s/
Disallow /*/c/*/*/
Disallow /*/c/*?*%2C*
Disallow /en/c/*?
Allow /de/c/o/*
Allow /*/c/*?
Disallow /*/shop/*/offers/*/
Disallow /*/shop/*/offers*?page=*&
Allow /*/shop/*/offers?page=
Allow /*/shop/*/offers/?page=
Disallow /*/shop/*/ratings*?
Disallow /*?*ar=
Disallow /*?*qcn=
Disallow /*?*nextOffset
Disallow /*?*next_offset
Disallow /*?*category=
Disallow /api/browser-statistics/
Disallow /api/frontend/search-autocomplete
Disallow /api/frontend/notifications
Disallow /api/frontend/categories/category-bar
Disallow /api/mfa/categories
Allow /api/mfa/categories/*/promo-offers
Disallow /api/mfa/notifications
Disallow /api/mfa/user/save-seller-id
Disallow /px/pv
Disallow /api/listing-form
Disallow /marketplace-spa/api/questions
Disallow /marketplace-spa/api/fee
Disallow /marketplace-spa/api/pdp
Allow /marketplace-spa/api/pdp/youtube
Disallow /static-assets/marketplace-spa/prod/_next/static/chunks/pages/_app-
Disallow /dataservice/
Disallow /pages/
Disallow /ajax/
Disallow /viewitem.aspx
Disallow /*feed.xml
Disallow /online-shop/
Disallow /shop/
Disallow /ratings/
Disallow /pages/*/fr.php
Disallow /auk/
Disallow /acheter/
Disallow /kaufen/
Disallow /zubehoer/
Disallow */v/
Disallow */w/
Disallow /cdn-cgi/

psbot
yandex
petalbot
mail.ru_bot
megaindex
baiduspider
yisouspider
bytespider
sogou web spider
sogou inst spider
proximic
admantx
ahrefs
oncrawl
seekport crawler
semrushbot
blexbot
mj12bot
zoombot
linkbot
gptbot
dotbot

Rule Path
Disallow /

adsbot-google-mobile
adsbot-google

Rule Path
Disallow /en/
Disallow /en$
Disallow /de/login
Disallow /fr/login
Disallow /it/login
Disallow /de/regist
Disallow /fr/regist
Disallow /it/regist
Disallow /*/bulk/landing
Disallow /*/my-ricardo
Disallow /*/bookkeeping
Disallow /de/profile
Disallow /fr/profile
Disallow /it/profile
Disallow /de/list
Disallow /fr/list
Disallow /it/list
Disallow /*/s$
Disallow /*/s/$
Disallow /*/s/*?
Disallow /*/shop/*/
Disallow /*?*sort=
Disallow /*?*ar=
Disallow /*?*qcn=
Disallow /*?*ignore_dominant=
Disallow /*?*nextOffset
Disallow /*?*next_offset
Disallow /*?*category=
Disallow /cdn-cgi/
Disallow /api/browser-statistics/
Disallow /api/frontend/search-autocomplete
Disallow /api/frontend/notifications
Disallow /api/frontend/categories
Allow /api/mfa/categories/*/promo-offers
Disallow /px/pv
Disallow /marketplace-spa/api/
Disallow /api/mfa/
Disallow /assets/search/
Disallow /api/listing-form
Disallow /static-assets/marketplace-spa/prod/_next/static/chunks/pages/_app-

criteobot/0.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0.2

grapeshot

Rule Path
Disallow
Back to top

Comments

  • robots.txt for https://www.ricardo.ch/
  • English pages until release
  • Don't try to index the login page, you will get a Cloudflare challenge
  • After login pages
  • SRP
  • CSRP
  • Sellers
  • Other params
  • Supporting files
  • Legacy
  • Disallow cloudflare /cdn-cgi/ endpoint.
  • See https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/
  • Disallow commercial bots we don't like
  • User agent names for Google AdsBot can be found here: https://support.google.com/webmasters/answer/1061943?hl=en
  • Limit criteo crawler to 5 req/sec max as it generates imaginary-wrapper traffic peaks
  • Allow Oracle Data Cloud Crawler
Back to top

Warnings

  • 1 invalid line.
Back to top