Log In Sign Up

magcraft.com
robots.txt

Robots Exclusion Standard data for magcraft.com

Resource Scan

Scan Details

Site Domain magcraft.com
Base Domain magcraft.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-18T01:58:05+00:00
Next Scan 2025-10-16T01:58:05+00:00

Last Successful Scan

Scanned2025-02-25T17:18:38+00:00
URL https://magcraft.com/robots.txt
Domain IPs 104.16.39.93, 104.16.40.93, 104.16.41.93, 104.16.42.93, 104.16.43.93, 2606:4700::6810:275d, 2606:4700::6810:285d, 2606:4700::6810:295d, 2606:4700::6810:2a5d, 2606:4700::6810:2b5d
Response IP 104.16.42.93
Found Yes
Hash f3252944898aaaabd91c286d40921d26fdba93300352ae1d6ee78ac429844c7e
SimHash bac14313e475

Groups

*

Product Comment
* These rules apply to all crawlers
Rule Path Comment
Disallow /store/admin/ Crawlers do not need access to your admin console, so this rule disallows all admin console pages for crawlers
Disallow /store/shopCart Crawlers cannot add items to a shopping cart, so prevent them from viewing the cart page
Disallow /store/shopCart.* Crawlers cannot add items to a shopping cart, so prevent them from viewing the cart page
Disallow /store/WriteR Crawlers cannot submit product reviews, so there is no need for them to view this page. The content of product reviews is on a separate page that is allowed
Disallow /store/addtocart AddToCart.aspx is used in embedded commerce and allows items to be added to the cart. There is no content on this page and therefore crawlers do not need to see it.
Disallow /store/OnePageCh generally contains no inherent SEO value, and crawlers cannot add items to their cart to be able to access this page, therefore it is disallowed
Disallow /store/checko This page contains no content
Disallow /*Attrib%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow /*?Attrib= If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow *Attribs%3D If you use attributes, these rules allow you to avoid duplicate content warnings when customers filter by specific attributes.
Disallow /highslide Block crawlers from accessing highslide urls, which is related to javascript used for viewing thumbnails
Disallow /blog?tags=* This rule allows you to avoid duplicate content warnings for blog content.
Disallow /store/adm Crawlers do not need access to your console, so this rule disallows all console pages for crawlers
Disallow /store/shopCa Crawlers cannot add items to a shopping cart, so prevent them from viewing this page
Disallow /store/addtoc is used in embedded commerce and allows items to be added to the cart. There is no content on this page and therefore crawlers do not need to see it.

yandex

Rule Path Comment
Disallow / Block Yandex crawler access to whole site
Back to top

Comments

  • This file allows you to control web crawlers access to specific pages on your site. Web crawlers are
  • programs that search engines run to view and analyze your site to index the content in their search engines.
  • Common crawlers include Googlebot and bingbot. These are the default rules defined for your site and include
  • pages and directories that crawlers do not need access to.
Back to top