Log In Sign Up

bfx.com.au
robots.txt

Robots Exclusion Standard data for bfx.com.au

Resource Scan

Scan Details

Site Domain bfx.com.au
Base Domain bfx.com.au
Scan Status Ok
Last Scan2024-10-30T21:18:58+00:00
Next Scan 2024-11-29T21:18:58+00:00

Last Scan

Scanned2024-10-30T21:18:58+00:00
URL https://bfx.com.au/robots.txt
Redirect https://www.bfx.com.au/robots.txt
Redirect Domain www.bfx.com.au
Redirect Base bfx.com.au
Domain IPs 151.101.2.132
Redirect IPs 151.101.2.132
Response IP 151.101.2.132
Found Yes
Hash 221143eb9031ed26800f31c246d44f193fbde879f15903303e16c2df289388f3
SimHash 37da5b0805f5

Groups

*
googlebot
googlebot-image

Rule Path
Allow /
Allow /store/*?id=*
Allow /wp-content/uploads/
Allow /wp-admin/admin-ajax.php
Allow /store/pub/media/
Allow /store/pub/static/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /readme.html
Disallow /refer/
Disallow /flipbooks/
Disallow /store/*shopby/*-*
Disallow /store/index.php/
Disallow /store/wishlist/
Disallow /store/admin/
Disallow /store/backbone/
Disallow /store/phpinfo.php
Disallow /store/enable-cookies/
Disallow /store/*?dir*
Disallow /store/*?dir=desc
Disallow /store/*?dir=asc
Disallow /store/*?limit=all
Disallow /store/*?mode*
Disallow /store/lib/
Disallow /store/*.php$
Disallow /store/pkginfo/
Disallow /store/report/
Disallow /store/var/
Disallow /store/catalog/
Disallow /store/sendfriend/
Disallow /store/review/
Disallow /store/*SID%3D
Disallow /store/*?
Disallow /store/checkout/
Disallow /store/quotation/
Disallow /store/onestepcheckout/
Disallow /store/catalogsearch/
Disallow /store/catalog/product_compare/
Disallow /store/catalog/category/view/
Disallow /store/catalog/product/view/
Disallow /store/app/
Disallow /store/bin/
Disallow /store/dev/
Disallow /store/lib/
Disallow /store/phpserver/
Disallow /store/pub/
Disallow /store/tag/
Disallow /store/review/
Disallow /store/composer.json
Disallow /store/composer.lock
Disallow /store/CONTRIBUTING.md
Disallow /store/CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /store/COPYING.txt
Disallow /store/Gruntfile.js
Disallow /store/LICENSE.txt
Disallow /store/LICENSE_AFL.txt
Disallow /store/nginx.conf.sample
Disallow /store/package.json
Disallow /store/php.ini.sample
Disallow /store/RELEASE_NOTES.txt
Disallow /store/*?*product_list_mode=
Disallow /store/*?*product_list_order=
Disallow /store/*?*product_list_limit=
Disallow /store/*?*product_list_dir=
Disallow /store/*.git
Disallow /store/*.CVS
Disallow /store/*.Zip$
Disallow /store/*.Svn$
Disallow /store/*.Idea$
Disallow /store/*.Sql$
Disallow /store/*.Tgz$
Back to top

Other Records

Field Value
sitemap https://www.bfx.com.au/sitemapindex.xml
Back to top

Comments

  • BFX Furniture robots.txt file updated by John 03 April 2024 ( added line 64 )
  • User Agents
  • Allow ALL Wordpress and Magento 2 directories
  • Allow Crawling of product variants for Google Merchant
  • Allow these Wordpress and Magento 2 directories
  • Disable these Wordpress directories
  • Disable the flipbooks directories
  • Disable Layered Navigation direcotories with more than one filter applied - Magento 2
  • Disable Magento home page so does not compete with Wordpress Home page - Magento 2
  • Disable these directories - Magento 2
  • Disallow these filter directories - Magento 2
  • Disable these common - Default Magento 2
  • Disable checkout & customer account - Default Magento 2
  • Disable Search pages - Default Magento 2
  • Disable common folders - Default Magento 2
  • Disable Tag & Review (Avoid duplicate content) - Default Magento 2
  • Common files - Default Magento 2
  • Disable sorting (Avoid duplicate content) - Default Magento 2
  • Disable version control folders and others - Default Magento 2
  • Sitemap
Back to top