Log In Sign Up

bulkpowders.com.au
robots.txt

Robots Exclusion Standard data for bulkpowders.com.au

Resource Scan

Scan Details

Site Domain bulkpowders.com.au
Base Domain bulkpowders.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-20T00:07:07+00:00
Next Scan 2026-04-20T00:07:07+00:00

Last Successful Scan

Scanned2022-12-10T12:42:23+00:00
URL https://bulkpowders.com.au/robots.txt
Redirect https://www.bulkpowders.com.au/robots.txt
Redirect Domain www.bulkpowders.com.au
Redirect Base bulkpowders.com.au
Domain IPs 104.26.0.214, 104.26.1.214, 172.67.71.108, 2606:4700:20::681a:1d6, 2606:4700:20::681a:d6, 2606:4700:20::ac43:476c
Redirect IPs 104.26.0.214, 104.26.1.214, 172.67.71.108, 2606:4700:20::681a:1d6, 2606:4700:20::681a:d6, 2606:4700:20::ac43:476c
Response IP 172.67.71.108
Found Yes
Hash 5eb20231089bd7b780ec0ac7908be0c12a4ae9b865f0bbf16c4f76a68048f64e
SimHash 2444ad326ff2

Groups

*

Rule Path
Allow /*?p=
Disallow */index.php/
Disallow */catalog/product_compare/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */catalog/product/gallery/
Disallow */catalogsearch/
Disallow */checkout/
Disallow */control/
Disallow */contacts/
Disallow */customer/
Disallow */customize/
Disallow */newsletter/
Disallow */poll/
Disallow */review/
Disallow */sales/
Disallow */sendfriend/
Disallow */tag/
Disallow */wishlist/
Disallow /*?p=*&
Disallow /*.php$
Disallow /*?SID=

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.bulkpowders.com.au/sitemap_bp_au.xml
sitemap https://www.bulkpowders.com.au/sitemap_wholesale.xml
sitemap https://www.naturalslim.com.au/sitemap_ns.xml
Back to top

Comments

  • For more information about the robots.txt standard visit
  • http://www.robotstxt.org/orig.html
  • For syntax checking visit
  • http://tool.motoricerca.info/robots-checker.phtml
  • Crawl the Sitemap. Set the correct URL before uncomment
  • Crawlers Setup
  • How many seconds a crawler should wait before loading and crawling page content
  • Set a custom crawl rate if you are experiencing traffic issues with your server
  • https://www.contentkingapp.com/academy/robotstxt/faq/crawl-delay-10/
  • Allow to crawl paging (paging inside a listing with more params are disallowed below)
  • Do not crawl non-SEF paths and generated content (if you use a store id in URL you must prefix with * or copy for each store)
  • Allow: */catalogsearch/seo_sitemap
  • Allow: */catalogsearch/term/popular
  • Do not crawl dynamic filters. Uncomment what you need or add custom filters
  • Disallow: /*?dir*
  • Disallow: /*?limit*
  • Disallow: /*?mode*
  • Disallow: /*?price=*
  • Disallow: /*?___from_store=*
  • Disallow: /*?___store=*
  • Disallow: /*?q=*
  • Disallow: /*?cat=*
  • Disallow: /*?availability=*
  • Disallow: /*?brand=*
  • Disallow: /*?
  • Do not crawl paths that can be safely ignored by search engines (no clean URLs)
  • Do not allow media indexing for the following bots
  • Disallow all or add custom paths. For example */media/ or */skin/
  • User-agent: baiduspider-image
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: baiduspider-video
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: msnbot-media
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: Googlebot-Image
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
  • User-agent: Googlebot-Video
  • Disallow: /
  • Disallow: */media/
  • Disallow: */skin/
Back to top