Log In Sign Up

kuberbox.com
robots.txt

Robots Exclusion Standard data for kuberbox.com

Resource Scan

Scan Details

Site Domain kuberbox.com
Base Domain kuberbox.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-13T15:09:39+00:00
Next Scan 2025-07-20T15:09:39+00:00

Last Successful Scan

Scanned2025-06-28T10:41:52+00:00
URL https://www.kuberbox.com/robots.txt
Domain IPs 151.101.130.137, 151.101.194.137, 151.101.2.137, 151.101.66.137, 2a04:4e42:200::649, 2a04:4e42:400::649, 2a04:4e42:600::649, 2a04:4e42::649
Response IP 146.75.46.137
Found Yes
Hash 105c22d8de6872d54562d71e7447ad6fd02b87040b5a8db6a45319294f8100b8
SimHash e6f2b101c641

Groups

*

Rule Path
Disallow /*.git$
Disallow /*.github$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /info.php
Disallow /admin_ufs69f/
Disallow /blog/wp-login
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*refcheck%3Drecos
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?cat*
Disallow /*?p*
Disallow /*?q*
Disallow /*?diamond_color=*
Disallow /*?diamond_clarity=*
Disallow /*?googleshopping_exclude=*
Disallow /*?ra_type=*
Disallow /*?refcheck=recos
Disallow /*?limit=30
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/errors/
Disallow /pub/opt/
Disallow /pub/static/
Disallow /tag/
Disallow /review/
Disallow /blog/tag/*
Disallow /index.php/
Disallow /*?SID=
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /searchtap/
Allow /*.js$
Allow /*.css$
Allow /blog/wp-content/plugins/*.js
Allow /blog/wp-content/plugins/*.css
Allow /blog/wp-content/themes/*.js
Allow /blog/wp-content/themes/*.css
Allow /blog/wp-content/cache/*.js
Allow /blog/wp-content/cache/*.css
Allow /blog/wp-includes/*.js

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

algoliarecommend/lookingsimilar

Rule Path
Disallow /

algolia accessibility crawler

Rule Path
Disallow /

algolia analytics crawler

Rule Path
Disallow /

algolia crawler

Rule Path
Disallow /

algolia docsearch crawler

Rule Path
Disallow /

algolia e-commerce crawler

Rule Path
Disallow /

algolia image crawler

Rule Path
Disallow /

algolia insights crawler

Rule Path
Disallow /

algolia inventory crawler

Rule Path
Disallow /

algolia local crawler

Rule Path
Disallow /

algolia media crawler

Rule Path
Disallow /

algolia mobile crawler

Rule Path
Disallow /

algolia news crawler

Rule Path
Disallow /

algolia personalization crawler

Rule Path
Disallow /

algolia places crawler

Rule Path
Disallow /

algolia product crawler

Rule Path
Disallow /

algolia recommend crawler

Rule Path
Disallow /

algolia security crawler

Rule Path
Disallow /

algolia seo crawler

Rule Path
Disallow /

algolia social crawler

Rule Path
Disallow /

algolia video crawler

Rule Path
Disallow /

looking similar crawler

Rule Path
Disallow /

amazonbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

meta-externalagent

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

gptbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

googleother

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.kuberbox.com/sitemap/sitemap.xml
sitemap https://www.kuberbox.com/sitemap/images_sitemap.xml
sitemap https://www.kuberbox.com/blog/sitemap.xml
Back to top

Comments

  • Website Sitemap - list all sitemaps here
  • Enable robots.txt rules for all crawlers
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • Do not crawl common server technical folders and files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Default Instructions
  • Disallow URL Filter Searches
  • add more filter options here
  • Restrict User Account and Checkout Pages
  • Restrict CMS Directories
  • Disallow Duplicate Content
  • Disallow Duplicate Blog Content
  • High overlap with blog landing page hence disallowed
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Disallow Catalog Search Pages
  • Allow all files ending with these extensions
  • Blog Scripts
  • Crawlers Setup
  • Crawl Delay Setup For Bots
  • updated on 220224 by SL
  • updated on 210324 by SL removed googlebot added refcheck recos filter
  • updated on 280924 by SL disallowed all dirs for meta bots due to excessive crawling
  • updated on 191024 by JC disallowed GPT Bots, set-up crawl delay due to excessive crawling
  • updated on 221024 by JC disallowed diamond color/clarity, ratype, refcheck, googleshopping and limit, many indexed pages formed due to this
  • updated on 111124 by JC disallowed GoogleOther due to excessive crawling
  • updated on 121124 by JC disallowed all algolia/lookingsimilar bots due to excessive crawling
  • updated on 140625 by JC allowed google,meta,chatgpt,apple bots with crawl delays
Back to top