cyberfit.com.au
robots.txt

Robots Exclusion Standard data for cyberfit.com.au

Resource Scan

Scan Details

Site Domain cyberfit.com.au
Base Domain cyberfit.com.au
Scan Status Ok
Last Scan2024-10-24T15:02:44+00:00
Next Scan 2024-11-23T15:02:44+00:00

Last Scan

Scanned2024-10-24T15:02:44+00:00
URL https://cyberfit.com.au/robots.txt
Redirect https://www.cyberfit.com.au/robots.txt
Redirect Domain www.cyberfit.com.au
Redirect Base cyberfit.com.au
Domain IPs 104.21.19.175, 172.67.187.7, 2606:4700:3031::ac43:bb07, 2606:4700:3034::6815:13af
Redirect IPs 104.21.19.175, 172.67.187.7, 2606:4700:3031::ac43:bb07, 2606:4700:3034::6815:13af
Response IP 104.21.19.175
Found Yes
Hash 4323a1209999a350b75df94f7a7c3cbfdd5d9e44c85586194c43cd9de0ec4cde
SimHash 8f2eb1333201

Groups

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow
Disallow /*?dir*
Disallow /*?limit=
Disallow /*?language=
Disallow /*?currency=
Disallow /*?mode*
Disallow /*?c=
Disallow /quickview/
Disallow /wishlist/
Disallow /catalog/
Disallow /amxsearchfront/
Disallow /ajaxcart/
Disallow /path/
Disallow /track/
Disallow /track-order/
Disallow *tel%3A*
Disallow /customer/account/login/referer/
Disallow /*?mgs*
Disallow /*?price*
Disallow /*?manufacturer*
Disallow /*?language*
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/

semrushbot-sa

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

yandex

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.cyberfit.com.au/pub/sitemap.xml

Comments

  • Do not crawl sub category pages that are sorted or filtered.
  • Don't crawl filters
  • Do not crawl seach pages and not-SEO optimized catalog links
  • Baiduspider
  • Yandex