Log In Sign Up

cleanflow.net
robots.txt

Robots Exclusion Standard data for cleanflow.net

Resource Scan

Scan Details

Site Domain cleanflow.net
Base Domain cleanflow.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-03-18T12:16:02+00:00
Next Scan 2026-04-17T12:16:02+00:00

Last Successful Scan

Scanned2026-02-10T10:54:33+00:00
URL https://cleanflow.net/robots.txt
Domain IPs 104.21.64.92, 172.67.180.140, 2606:4700:3030::ac43:b48c, 2606:4700:3034::6815:405c
Response IP 104.21.64.92
Found Yes
Hash 588016b1a4282f56f96b1d6f9293d2f6f399cbf31c9732c79f1db4f1620389e0
SimHash 6759c950cefd

Groups

*

Rule Path
Disallow /a/downloads/-/*
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /10165405/checkouts
Disallow /10165405/orders
Disallow /account
Disallow /apps/*
Disallow /tools/*
Disallow /discount/*
Disallow */collections/*/products/*
Disallow /search
Disallow /*?q=*
Disallow /*%26q%3D*
Disallow /*?page=*
Disallow /*%26page%3D*
Disallow /collections/*?sort_by=*
Disallow /*/collections/*?sort_by=*
Disallow /collections/*/*tagged*
Disallow /*/collections/*/*tagged*
Disallow /*?filter.*
Disallow /*%26filter.*
Disallow /collections/*filter*%26*filter*
Disallow /*/collections/*filter*%26*filter*
Disallow /*?pf_*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /blogs/*/tagged/
Disallow /*?variant=*&*
Disallow /*%26variant%3D*
Disallow /*?constraint=*
Disallow /*?selling_plan=*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /*?pr_prod_strat*
Disallow /*?pr_rec_id*
Disallow /*?pr_rec_pid*
Disallow /*?pr_ref_pid*
Disallow /*?pr_seq=*
Disallow /*_rdiscovery*
Disallow /*/*?*ls=*&ls=*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /*?_pos=*
Disallow /*?_psq=*
Disallow /*?_ss=*
Disallow /*?_v=*
Disallow /*?utm_*
Disallow /*?fbclid=*
Disallow /*?gclid=*
Disallow /*?msclkid=*
Disallow /*?_ga=*
Disallow /*?_gl=*
Disallow /*?ref=*
Disallow /*?mc_cid=*
Disallow /*?mc_eid=*
Disallow /*?contact_posted=*
Disallow /*?customer_posted=*
Disallow /*?locale=*
Disallow /*?currency=*
Disallow /*.atom
Disallow /*.atom?*
Disallow /apple-app-site-association
Disallow /.well-known/shopify/monorail
Disallow /cdn/wpm/*.js
Disallow /wpm%40*
Disallow /recommendations/
Disallow /*/recommendations/products
Disallow /*-remote
Disallow /cdn/shop/files/*
Allow /cdn/shop/files/*.pdf
Allow /cdn/shop/files/*.jpg
Allow /cdn/shop/files/*.jpeg
Allow /cdn/shop/files/*.png
Allow /cdn/shop/files/*.webp
Allow /cdn/shop/files/*.gif
Allow /policies/
Allow /*/policies/
Allow /collections/
Allow /products/

googlebot
adsbot-google
googlebot-image
googlebot-news

Rule Path
Allow /checkouts/
Allow /checkout
Allow /carts
Allow /10165405/checkouts
Allow /products/*?variant=*
Disallow /admin
Disallow /cart
Disallow /orders
Disallow /10165405/orders
Disallow /account
Disallow /a/downloads/-/*
Disallow /apps/*
Disallow /tools/*
Disallow /discount/*
Disallow */collections/*/products/*
Disallow /search
Disallow /*?q=*
Disallow /*%26q%3D*
Disallow /*?page=*
Disallow /*%26page%3D*
Disallow /collections/*?sort_by=*
Disallow /*/collections/*?sort_by=*
Disallow /collections/*/*tagged*
Disallow /*/collections/*/*tagged*
Disallow /*?filter.*
Disallow /*%26filter.*
Disallow /collections/*filter*%26*filter*
Disallow /*/collections/*filter*%26*filter*
Disallow /*?pf_*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /blogs/*/tagged/
Disallow /*?variant=*&*
Disallow /*%26variant%3D*
Disallow /*?constraint=*
Disallow /*?selling_plan=*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /*?pr_prod_strat*
Disallow /*?pr_rec_id*
Disallow /*?pr_rec_pid*
Disallow /*?pr_ref_pid*
Disallow /*?pr_seq=*
Disallow /*_rdiscovery*
Disallow /*/*?*ls=*&ls=*
Disallow /*?_pos=*
Disallow /*?_psq=*
Disallow /*?_ss=*
Disallow /*?_v=*
Disallow /*?utm_*
Disallow /*?fbclid=*
Disallow /*?gclid=*
Disallow /*?msclkid=*
Disallow /*?_ga=*
Disallow /*?_gl=*
Disallow /*?ref=*
Disallow /*?mc_cid=*
Disallow /*?mc_eid=*
Disallow /*?contact_posted=*
Disallow /*?customer_posted=*
Disallow /*?locale=*
Disallow /*?currency=*
Disallow /*.atom
Disallow /*.atom?*
Disallow /apple-app-site-association
Disallow /.well-known/shopify/monorail
Disallow /cdn/wpm/*.js
Disallow /wpm%40*
Disallow /recommendations/
Disallow /*/recommendations/products
Disallow /*-remote
Disallow /cdn/shop/files/*
Allow /cdn/shop/files/*.pdf
Allow /cdn/shop/files/*.jpg
Allow /cdn/shop/files/*.jpeg
Allow /cdn/shop/files/*.png
Allow /cdn/shop/files/*.webp
Allow /cdn/shop/files/*.gif
Allow /policies/
Allow /*/policies/
Allow /collections/
Allow /products/

bingbot
msnbot

Rule Path
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /account
Disallow /apps/*
Disallow /tools/*
Disallow /discount/*
Disallow */collections/*/products/*
Disallow /search
Disallow /*?q=*
Disallow /*%26q%3D*
Disallow /*?page=*
Disallow /*%26page%3D*
Disallow /collections/*?sort_by=*
Disallow /*/collections/*?sort_by=*
Disallow /collections/*/*tagged*
Disallow /*/collections/*/*tagged*
Disallow /*?filter.*
Disallow /*%26filter.*
Disallow /collections/*filter*%26*filter*
Disallow /*/collections/*filter*%26*filter*
Disallow /*?pf_*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*?variant=*&*
Disallow /*%26variant%3D*
Disallow /*?constraint=*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /*?pr_prod_strat*
Disallow /*_rdiscovery*
Disallow /*?utm_*
Disallow /*?fbclid=*
Disallow /*?gclid=*
Disallow /*?msclkid=*
Disallow /*?_ga=*
Disallow /*?_gl=*
Disallow /*?_pos=*
Disallow /*?_psq=*
Disallow /*?_ss=*
Disallow /*?_v=*
Disallow /*.atom
Disallow /recommendations/
Disallow /*-remote

ahrefsbot

Rule Path
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /account
Disallow /apps/*
Disallow /search
Disallow /*?q=*
Disallow /*?page=*
Disallow /*?filter.*
Disallow /*.atom
Disallow /recommendations/
Disallow */collections/*/products/*

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /account
Disallow /apps/*
Disallow /search
Disallow /*?q=*
Disallow /*?page=*
Disallow /*?filter.*
Disallow /*.atom
Disallow /recommendations/
Disallow */collections/*/products/*

Other Records

Field Value
crawl-delay 10

screaming frog seo spider

Rule Path
Allow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

nutch

Rule Path
Disallow /

gptbot

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

google-extended

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

anthropic-ai

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

claudebot

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

perplexitybot

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

applebot-extended

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

ccbot

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*

meta-externalagent

Rule Path
Allow /
Disallow /admin
Disallow /cart
Disallow /checkout
Disallow /account
Disallow /apps/*
Back to top

Other Records

Field Value
sitemap https://cleanflow.net/sitemap.xml
sitemap https://cleanflow.net/sitemap.xml
sitemap https://cleanflow.net/sitemap.xml
sitemap https://cleanflow.net/sitemap.xml
sitemap https://cleanflow.net/sitemap.xml
Back to top

Comments

  • =============================================================================
  • CleanFlow.net robots.txt (Static Optimized v2.1)
  • Last Updated: January 2025
  • =============================================================================
  • =============================================================================
  • CONTENT SIGNALS
  • =============================================================================
  • As a condition of accessing this website, you agree to abide by the following:
  • - search=yes: You may collect content for search index purposes
  • - ai-train=no: You may not collect content for AI training purposes
  • --- Core Shopify System Pages ---
  • --- Duplicate Product URLs (Critical for Canonicalization) ---
  • Blocks /collections/all/products/x and /collections/name/products/x
  • --- Search & Pagination ---
  • --- Collection Sorting & Filtering ---
  • --- Tag Combinations (+ and URL-encoded variants) ---
  • --- Variant & Product Parameters ---
  • --- Shopify Tracking & System Parameters ---
  • --- Shopify Predictive Search Parameters ---
  • --- Marketing & Analytics Tracking Parameters ---
  • --- Form Confirmation Parameters ---
  • --- Locale & Currency Parameters ---
  • --- Feed & System Files ---
  • --- Remote/Generated Product URLs ---
  • --- CDN Files (Block system files, Allow assets) ---
  • --- Explicitly Allow Important Sections ---
  • =============================================================================
  • GOOGLEBOT (Search Crawlers)
  • =============================================================================
  • Googlebot needs explicit rules - it does not inherit from User-agent: *
  • when a specific Googlebot block exists
  • --- Allow for Google Ads Conversion Tracking ---
  • --- Allow Product Variants for GMC ---
  • --- Core Blocked Areas ---
  • --- Duplicate Product URLs (Critical) ---
  • --- Search & Pagination ---
  • --- Collection Sorting & Filtering ---
  • --- Tag Combinations ---
  • --- Variant & Product Parameters ---
  • --- Shopify Tracking & System Parameters ---
  • --- Shopify Predictive Search Parameters ---
  • --- Marketing & Analytics Tracking Parameters ---
  • --- Form Confirmation Parameters ---
  • --- Locale & Currency Parameters ---
  • --- Feed & System Files ---
  • --- Remote/Generated Product URLs ---
  • --- CDN Files ---
  • --- Explicitly Allow ---
  • =============================================================================
  • BING (Search Crawlers)
  • =============================================================================
  • --- Core Blocked Areas ---
  • --- Duplicate Product URLs ---
  • --- Search & Pagination ---
  • --- Collection Sorting & Filtering ---
  • --- Tag Combinations ---
  • --- Variant & Product Parameters ---
  • --- Tracking Parameters ---
  • --- Predictive Search ---
  • --- Feed & System Files ---
  • =============================================================================
  • SEO TOOLS (Allow with Rate Limiting)
  • =============================================================================
  • =============================================================================
  • BLOCKED CRAWLERS (AI Training / Scraping)
  • =============================================================================
  • =============================================================================
  • AI ASSISTANTS (Allow for RAG/Grounding - Not Training)
  • =============================================================================
  • These bots retrieve content for real-time AI responses
  • Content-signal above prohibits use for training
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top