Log In Sign Up

medias.schaeffler.de
robots.txt

Robots Exclusion Standard data for medias.schaeffler.de

Resource Scan

Scan Details

Site Domain medias.schaeffler.de
Base Domain schaeffler.de
Scan Status Ok
Last Scan2024-09-26T13:23:44+00:00
Next Scan 2024-10-26T13:23:44+00:00

Last Scan

Scanned2024-09-26T13:23:44+00:00
URL https://medias.schaeffler.de/robots.txt
Domain IPs 20.73.43.66
Response IP 20.73.43.66
Found Yes
Hash 56dbbf416661dbb4e59e2f1997ce80480a514af0d32c04c772c7079927a82ac0
SimHash 2457df14eceb

Groups

*

Rule Path
Disallow /*/cart/
Disallow /*/checkout/
Disallow /*/my-account/
Disallow /*/wishlist/
Disallow /*/search/
Disallow /*/my-company/
Disallow /*/abmonitor/
Disallow /*/p/pdf/

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

mozilla/5.0 (compatible; yandexbot/3.0; +http://yandex.com/bots)

Rule Path
Disallow /

mozilla/5.0 (compatible; yandexbot/3.0; +http://yandex.com/bots) applewebkit/537.36 (khtml, like gecko) chrome/108.0.0.0

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://medias.schaeffler.de/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • CS20210001104901
  • Block Baiduspider
  • Block Baiduspider
  • Block Baiduspider
  • Block Baiduspider
  • Block Baiduspider
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block Yandex
  • Block YandexBot
  • Block FacebookBot
  • Block FacebookBot
Back to top