vaporwarehouse.com
robots.txt

Robots Exclusion Standard data for vaporwarehouse.com

Resource Scan

Scan Details

Site Domain vaporwarehouse.com
Base Domain vaporwarehouse.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-20T18:28:17+00:00
Next Scan 2026-01-18T18:28:17+00:00

Last Successful Scan

Scanned2025-03-01T19:36:53+00:00
URL https://www.vaporwarehouse.com/robots.txt
Domain IPs 104.16.39.93, 104.16.40.93, 104.16.41.93, 104.16.42.93, 104.16.43.93, 2606:4700::6810:275d, 2606:4700::6810:285d, 2606:4700::6810:295d, 2606:4700::6810:2a5d, 2606:4700::6810:2b5d
Response IP 104.16.40.93
Found Yes
Hash 8a097fec7668c9f1a4fa0963e149b6c5f99ca73280bf9166abdb8ab59f21fc41
SimHash 34e00353c5f5

Groups

*

Rule Path Comment
Disallow /store/profile/ -
Disallow /blog/2005_blog/04/5-year-warranties-on-vapor-warez.html -
Disallow /site/contactold.html -
Disallow /store/shopcart.aspx -
Disallow /store/ShopCart.aspx -
Disallow /store/login.aspx -
Disallow /store/TellAFriend.aspx -
Disallow /store/ExplainMAP.aspx -
Disallow /reviews/vb.html -
Disallow /Shared/Themes/ Prevents crawlers from viewing your theme markup files. These files contain no inherent SEO value
Disallow /Shared/CSS/ Prevents crawlers from viewing your CSS files. These files contain no inherent SEO value
Disallow /store/inc/ Prevents crawlers from viewing system Javascript. These files contain no inherent SEO value
Disallow /store/adm Crawlers do not need access to your console, so this rule disallows all console pages for crawlers
Disallow /store/shopCa Crawlers cannot add items to a shopping cart, so prevent them from viewing the cart page
Disallow /store/WriteR Crawlers cannot submit product reviews, so there is no need for them to view this page. The content of product reviews is on a separate page that is allowed
Disallow /store/addtoc is used in embedded commerce and allows items to be added to the cart. There is no content on this page and therefore crawlers do not need to see it.
Disallow /store/OnePageCh generally contains no inherent SEO value, and crawlers cannot add items to their cart to be able to access this page, therefore it is disallowed
Disallow /store/checko This page contains no content
Disallow /*Attrib%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow /*?Attrib= If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *Attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.

amazonbot

Product Comment
amazonbot Amazon's user agent
Rule Path
Disallow /

googlebot

Rule Path Comment
Disallow /blog -
Disallow /store/blog/1-Vaporizer-News-and-Info-by-Vapor-Warehouse.html -
Disallow /store/shopcart.aspx -
Disallow /store/ShopCart.aspx -
Disallow /store/login.aspx -
Disallow /store/TellAFriend.aspx -
Disallow /store/ExplainMAP.aspx -
Disallow /reviews/vb.html -
Disallow /emaildirect-boxed.html -
Disallow /emaildirect-inline.html -
Disallow /store/ProductReviews.aspx?ProductID=37 -
Disallow /store/ProductReviews.aspx?ProductID=604 -
Disallow /store/adm Crawlers do not need access to your console, so this rule disallows all console pages for crawlers
Disallow /store/shopCa Crawlers cannot add items to a shopping cart, so prevent them from viewing the cart page
Disallow /store/WriteR Crawlers cannot submit product reviews, so there is no need for them to view this page. The content of product reviews is on a separate page that is allowed
Disallow /store/addtoc is used in embedded commerce and allows items to be added to the cart. There is no content on this page and therefore crawlers do not need to see it.
Disallow /store/OnePageCh generally contains no inherent SEO value, and crawlers cannot add items to their cart to be able to access this page, therefore it is disallowed
Disallow /store/checko This page contains no content
Disallow /*Attrib%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow /*?Attrib= If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *Attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Allow /store/admin/inc/vendor/jquery.js -
Allow /store/admin/inc/scripts/ac.productgroups.js -
Allow /store/inc/style/jquery.ui.autocomplete.css -
Allow /Shared/Themes/* -

Comments

  • Disallow: /store/ProductReviews.aspx # block product reviews till we can get unique descriptions
  • User-agent: Exabot
  • Crawl-delay: 360
  • allow googlebot to access css and javascript for mobile display verification but still prevent other stuff
  • block reviews that have static URLs to prevent duplicate content