Log In Sign Up

now.breakthrubev.com
robots.txt

Robots Exclusion Standard data for now.breakthrubev.com

Resource Scan

Scan Details

Site Domain now.breakthrubev.com
Base Domain breakthrubev.com
Scan Status Ok
Last Scan2025-09-30T00:46:35+00:00
Next Scan 2025-10-30T00:46:35+00:00

Last Scan

Scanned2025-09-30T00:46:35+00:00
URL https://now.breakthrubev.com/robots.txt
Domain IPs 151.101.131.52, 151.101.195.52, 151.101.3.52, 151.101.67.52
Response IP 199.232.47.52
Found Yes
Hash 4d856dca2b3c51bab57ee2d4f0c7fccdb1aa4945c742bd02a87c4a5d0d091290
SimHash ec405f5eedf0

Groups

*

Rule Path
Allow /bbg/en/login
Disallow /
Back to top

Comments

  • For all robots
  • User-agent: *
  • Block access to specific groups of pages
  • Disallow: /bbg/en/cart
  • Disallow: /bbg/en/checkout
  • Disallow: /bbg/en/my-account
  • Request-rate: 1/10 # maximum rate is one page every 10 seconds
  • Crawl-delay: 10 # 10 seconds between page requests
  • Visit-time: 0400-0845 # only visit between 04:00 and 08:45 UTC
  • Allow search crawlers to discover the sitemap
  • Sitemap: /bbg/en/sitemap.xml
  • Block CazoodleBot as it does not present correct accept content headers
  • User-agent: CazoodleBot
  • Disallow: /
  • Block MJ12bot as it is just noise
  • User-agent: MJ12bot
  • Disallow: /
  • Block dotbot as it cannot parse base urls properly
  • User-agent: dotbot/1.0
  • Disallow: /
  • Block Gigabot
  • User-agent: Gigabot
  • Disallow: /
Back to top