Log In Sign Up

herbano.com
robots.txt

Robots Exclusion Standard data for herbano.com

Resource Scan

Scan Details

Site Domain herbano.com
Base Domain herbano.com
Scan Status Ok
Last Scan2024-05-25T02:52:38+00:00
Next Scan 2024-06-24T02:52:38+00:00

Last Scan

Scanned2024-05-25T02:52:38+00:00
URL https://herbano.com/robots.txt
Domain IPs 104.21.63.39, 172.67.169.116, 2606:4700:3034::6815:3f27, 2606:4700:3035::ac43:a974
Response IP 172.67.169.116
Found Yes
Hash 067b3fac9a24fbc988087925eac45c7f37c8a3014595d25bf973c5b92e0651ab
SimHash 09217251c663

Groups

*

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /lp/
Disallow /lp*
Disallow /admin/
Disallow /admin*
Disallow /store1/
Disallow /store2/
Disallow /store3/
Disallow /store1*
Disallow /store2*
Disallow /store3*
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /www/
Disallow /api.php
Disallow /cron.php
Disallow /cronjobs
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=
Disallow /*?limit=10
Disallow /*?limit=20
Disallow /*?limit=50
Disallow /*?redirect
Disallow /*?mode
Disallow /*ajax
Disallow /*?price
Disallow /*?order
Disallow /*?sex_target
Disallow /*?occasions_sowia
Disallow /*?gender
Disallow /*?personalization
Disallow /*?linktype
Disallow /*?message
Disallow /*?utm_source=
Disallow /*%26utm_medium%3D
Disallow /*%26utm_campaign%3D
Disallow /*?sid=
Disallow /*?gclid=
Disallow /service-unavailable
Disallow /*?page=
Disallow /*?page=2
Disallow /media/pdf
Disallow /*?p=
Disallow /*?p=4
Disallow /*?p=2
Disallow /*?p=3
Disallow /*?limit=10&p=2
Disallow /*?limit=20&p=2
Disallow /*?limit=10&p=3
Allow /static/*.svg
Allow /static/*.js
Allow /static/*.css
Allow /static/*.png
Allow /static/*.jpg
Allow /static/*.jpeg
Disallow /*?SID=
Disallow /checkout/
Disallow /de/checkout/
Disallow /ch/checkout/
Disallow /at/checkout/
Disallow /fr/checkout/
Disallow /it/checkout/
Disallow /nl_BE/checkout/
Disallow /fr_BE/checkout/
Disallow /nl/checkout/
Disallow /onestepcheckout/
Disallow /mycheckout/
Disallow /customer/
Disallow /de/customer/
Disallow /ch/customer/
Disallow /at/customer/
Disallow /fr/customer/
Disallow /it/customer/
Disallow /nl_BE/customer/
Disallow /fr_BE/customer/
Disallow /nl/customer/
Disallow /customer/account/
Disallow /de/customer/account/
Disallow /ch/customer/account/
Disallow /at/customer/account/
Disallow /fr/customer/account/
Disallow /it/customer/account/
Disallow /nl_BE/customer/account/
Disallow /fr_BE/customer/account/
Disallow /nl/customer/account/
Disallow /customer/account/login/
Disallow /de/customer/account/login/
Disallow /ch/customer/account/login/
Disallow /at/customer/account/login/
Disallow /fr/customer/account/login/
Disallow /it/customer/account/login/
Disallow /nl_BE/customer/account/login/
Disallow /fr_BE/customer/account/login/
Disallow /nl/customer/account/login/
Disallow /checkout/cart/index/
Disallow /de/checkout/cart/index/
Disallow /ch/checkout/cart/index/
Disallow /at/checkout/cart/index/
Disallow /fr/checkout/cart/index/
Disallow /it/checkout/cart/index/
Disallow /nl_BE/checkout/cart/index/
Disallow /fr_BE/checkout/cart/index/
Disallow /nl/checkout/cart/index/
Disallow /checkout/cart/
Disallow /de/checkout/cart/
Disallow /ch/checkout/cart/
Disallow /at/checkout/cart/
Disallow /fr/checkout/cart/
Disallow /it/checkout/cart/
Disallow /nl_BE/checkout/cart/
Disallow /fr_BE/checkout/cart/
Disallow /nl/checkout/cart/
Disallow /catalogsearch/
Disallow /catalogsearch/result/
Disallow /catalog/product_compare/
Disallow /de/catalog/product_compare/
Disallow /fr/catalog/product_compare/
Disallow /ch/catalog/product_compare/
Disallow /at/catalog/product_compare/
Disallow /it/catalog/product_compare/
Disallow /nl_BE/catalog/product_compare/
Disallow /fr_BE/catalog/product_compare/
Disallow /nl/catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /review/product/
Disallow /stockalert/index/
Disallow /activebanner/
Disallow /ads/
Disallow /images/
Disallow /newsletter/
Disallow /krone/
Disallow /wishlist/
Disallow /de/wishlist/
Disallow /at/wishlist/
Disallow /ch/wishlist/
Disallow /fr/wishlist/
Disallow /it/wishlist/
Disallow /nl_BE/wishlist/
Disallow /fr_BE/wishlist/
Disallow /nl/wishlist/
Disallow /giftsearch_scripts/
Disallow /tag/
Disallow /captcha/refresh/
Disallow /fr/ratgeber/
Disallow /it/ratgeber/
Disallow /nl_BE/ratgeber/
Disallow /fr_BE/ratgeber/
Disallow /nl/ratgeber/
Disallow /de/magazine/
Disallow /ch/magazine/
Disallow /at/magazine/
Disallow /nl_BE/magazine/
Disallow /nl/magazine/
Disallow /de/raadsman/
Disallow /at/raadsman/
Disallow /ch/raadsman/
Disallow /fr/raadsman/
Disallow /it/raadsman/
Disallow /fr_BE/raadsman/
Disallow /fr/produits/
Disallow /de/produkte/
Disallow /ch/produkte/
Disallow /at/produkte/
Disallow /it/tutti-i-prodotti/
Disallow /nl_BE/producten/
Disallow /fr_BE/produits/
Disallow /nl/producten/
Disallow /stores/store/redirect/
Disallow /stores/store/switch/
Disallow /ratgeber?p=
Disallow /?linktype=recommended-product
Disallow /*?linktype=recommended-product
Disallow /*?linktype=recommended-product-widget
Disallow /*?linktype=homepage-banner
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Back to top

Other Records

Field Value
sitemap https://herbano.com/sitemap.xml
sitemap https://herbano.com/at/sitemap.xml
sitemap https://herbano.com/ch/sitemap.xml
sitemap https://herbano.com/fr/sitemap.xml
sitemap https://herbano.com/it/sitemap.xml
sitemap https://herbano.com/nl_BE/sitemap.xml
sitemap https://herbano.com/fr_BE/sitemap.xml
sitemap https://herbano.com/nl/sitemap.xml
Back to top

Comments

  • robots.txt for Magento Community and Enterprise
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump filess
  • Do not crawl Magento admin page
  • Do not crawl Magento product landing page
  • Do not crawl store1, store2, store3
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl search pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • User-agent: msnbot-media
  • Disallow: /
Back to top