totaltools.com.au
robots.txt

Robots Exclusion Standard data for totaltools.com.au

Resource Scan

Scan Details

Site Domain totaltools.com.au
Base Domain totaltools.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-04T18:37:09+00:00
Next Scan 2025-07-18T18:37:09+00:00

Last Successful Scan

Scanned2025-05-27T18:36:42+00:00
URL https://totaltools.com.au/robots.txt
Redirect https://www.totaltools.com.au/robots
Redirect Domain www.totaltools.com.au
Redirect Base totaltools.com.au
Domain IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Redirect IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Response IP 146.75.45.124
Found Yes
Hash d0a5ac665d41fd13bcbc8cc1feae538590aeda24f511e2a9e4f7627992ef6f2e
SimHash c1184942ad93

Groups

*

Rule Path
Disallow /ajax/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product_compare/
Disallow /checkout/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /productlist/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /app-home
Disallow /app-offers
Disallow /storelocator/index/index/
Disallow /request-information/forgot-password/
Disallow /request-information/sign-up/
Disallow /request-information/logout/
Disallow /search/
Disallow /*?CMP
Disallow /*%26CMP
Disallow /*?int_cmp
Disallow /*%26int_cmp

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Disallow /*.php*
Disallow /*.sh*
Disallow /*?
Disallow /*SID%3D
Allow /*?p=
Allow /static/
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.jpeg
Allow /*.png
Allow /*.gif

siteauditbot

Rule Path
Disallow /careers

semrushbot-si

Rule Path
Disallow /careers

Other Records

Field Value
sitemap https://www.totaltools.com.au/sitemap.xml
sitemap https://www.totaltools.com.au/media/sitemap_product_product.xml
sitemap https://www.totaltools.com.au/media/sitemap_category.xml
sitemap https://www.totaltools.com.au/media/sitemap_additional_link.xml
sitemap https://www.totaltools.com.au/media/sitemap_cms.xml
sitemap https://www.totaltools.com.au/media/sitemap_other.xml

Comments

  • Directory Excludes
  • Block Global Parameters
  • Disallow directives for AI web crawlers