firepitsdirect.com
robots.txt

Robots Exclusion Standard data for firepitsdirect.com

Resource Scan

Scan Details

Site Domain firepitsdirect.com
Base Domain firepitsdirect.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-08-28T21:14:01+00:00
Next Scan 2025-11-26T21:14:01+00:00

Last Successful Scan

Scanned2025-04-08T19:20:20+00:00
URL https://firepitsdirect.com/robots.txt
Redirect http://www.firepitsdirect.com/robots/fpd/robots.txt
Redirect Domain www.firepitsdirect.com
Redirect Base firepitsdirect.com
Domain IPs 104.21.41.92, 172.67.190.164, 2606:4700:3033::6815:295c, 2606:4700:3037::ac43:bea4
Redirect IPs 104.21.41.92, 172.67.190.164, 2606:4700:3033::6815:295c, 2606:4700:3037::ac43:bea4
Response IP 104.21.41.92
Found Yes
Hash d094a05ce1844c55c51db670ce94c8d4cb1159c2a850a94d2fadd1548225cefe
SimHash 6d24793542f1

Groups

beijing

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

brandverity

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /admin/
Disallow /admin_*/
Disallow /rest/
Disallow /page_cache/block/
Disallow /cache_warmer/
Disallow /spotixcache/
Disallow /404/
Disallow /manage/
Disallow /report/
Disallow /stats/
Disallow /control/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /var/
Disallow /contacts/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /sendfriend/
Disallow /wishlist/
Disallow /manuals/
Disallow /warmer/
Disallow /magmimagmi/
Disallow /feeds/
Disallow .sql
Disallow .tgz
Disallow /questions/
Disallow /review/product/list/id/
Disallow /amconf/ajax/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /health_check.php
Disallow /get.php
Disallow /*?amp=
Disallow /*?product_list_dir=
Disallow /*?product_list_limit=
Disallow /*?product_list_mode=
Disallow /*-filter--*
Disallow /*-filter%3D*
Disallow /*_filter--*
Disallow /*_filter%3D*
Disallow /*?dir=
Disallow /*?limit=
Disallow /*?onSale=
Disallow /*?type=
Disallow /*?p=
Disallow /*?pp=
Disallow /*?page=
Disallow /*?___from_store=
Disallow /*?___store=
Disallow /*?rating=
Disallow /*?stock=
Disallow /index.php
Disallow /*SID%3D
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /sales/billing_agreement/
Disallow /sales/recurring_profile/
Disallow /sales/order/history/
Disallow /sales/order/view/
Disallow /blog/search/
Disallow /searchspring/
Disallow /nsearch
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /tag/
Disallow /modsec-shared-collections-lock
Disallow /modsec-shared-collections
Disallow /warp.php
Disallow *manufacturer--*
Disallow *manufacturer%3D*

Other Records

Field Value
sitemap https://www.firepitsdirect.com/pub/sitemaps/fpd.xml

Comments

  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/)
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot-Image
  • Disallow: /
  • User-agent: msnbot-media
  • Disallow: /
  • Manufacturers