hvalleytools.com
robots.txt

Robots Exclusion Standard data for hvalleytools.com

Resource Scan

Scan Details

Site Domain hvalleytools.com
Base Domain hvalleytools.com
Scan Status Ok
Last Scan2026-02-11T05:40:13+00:00
Next Scan 2026-03-13T05:40:13+00:00

Last Scan

Scanned2026-02-11T05:40:13+00:00
URL https://hvalleytools.com/robots.txt
Redirect https://www.hvalleytools.com/robots.txt
Redirect Domain www.hvalleytools.com
Redirect Base hvalleytools.com
Domain IPs 104.26.12.173, 104.26.13.173, 172.67.74.209, 2606:4700:20::681a:cad, 2606:4700:20::681a:dad, 2606:4700:20::ac43:4ad1
Redirect IPs 104.26.12.173, 104.26.13.173, 172.67.74.209, 2606:4700:20::681a:cad, 2606:4700:20::681a:dad, 2606:4700:20::ac43:4ad1
Response IP 104.26.13.173
Found Yes
Hash 5352ef9e7a1193ebe43ba543050453403a42aa38674dadf746787069283577bc
SimHash ad80a95706e3

Groups

*

Rule Path
Allow /
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /setup/
Disallow /update/
Disallow /var/
Disallow /vendor/
Disallow /admin/
Disallow /admin_*
Disallow /index.php/admin/
Disallow /index.php/backend/
Disallow /composer.json
Disallow /composer.lock
Disallow /auth.json
Disallow /package.xml
Disallow /php.ini
Disallow /.user.ini
Disallow /CHANGELOG.txt
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.md
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /errors/
Disallow /includes/
Disallow /downloader/
Disallow /pkginfo/
Disallow /shell/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /error_log
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /sales/
Disallow /wishlist/
Disallow /sendfriend/
Disallow /review/
Disallow /newsletter/
Disallow /contact/
Disallow /ajax/
Disallow /rest/
Disallow /graphql/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /terms-of-sale/
Disallow /terms-of-use/
Disallow /privacy-policy/
Disallow /sales/order/
Disallow /storecredit/info/
Disallow /invitation/
Disallow /*?SID=
Disallow /*?dir=
Disallow /*?limit=
Disallow /*?mode=
Disallow /*?price=
Disallow /*?cat=
Disallow /*?color=
Disallow /*?size=
Disallow /*?manufacturer=
Disallow /*?q=
Disallow /*?p=
Disallow /*?product_list_order=
Disallow /*?product_list_mode=
Disallow /*?product_list_limit=
Disallow /*?utm_*
Disallow /*?fbclid=*
Disallow /*?gclid=*
Disallow */index.php/
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /CVS
Allow /static/
Allow /media/
Allow /*.js$
Allow /*.css$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Allow /cdn-cgi/
Allow /cdn-cgi/image/
Allow /cdn-cgi/l/email-protection

Comments

  • Robots.txt – All Domains (Magento 2.4)
  • Optimized for SEO, Security & Cloudflare
  • --- Allow search engines to index public pages ---
  • --- Core Magento folders to keep private ---
  • --- Admin / backend (update if custom path) ---
  • --- Sensitive system/config files ---
  • --- Technical & server files ---
  • --- Dynamic & customer pages (not for indexing) ---
  • --- Non-SEO CMS or policy pages ---
  • --- Query-parameter junk (sorting, filters, tracking) ---
  • --- Prevent crawling of duplicate/index.php URLs ---
  • --- Development / IDE / dump files ---
  • --- Allow static & media for rendering ---
  • --- Cloudflare CDN compatibility ---