Log In Sign Up

bestandless.com.au
robots.txt

Robots Exclusion Standard data for bestandless.com.au

Resource Scan

Scan Details

Site Domain bestandless.com.au
Base Domain bestandless.com.au
Scan Status Ok
Last Scan2025-12-30T08:26:04+00:00
Next Scan 2026-01-13T08:26:04+00:00

Last Scan

Scanned2025-12-30T08:26:04+00:00
URL https://bestandless.com.au/robots.txt
Redirect https://www.bestandless.com.au/robots.txt
Redirect Domain www.bestandless.com.au
Redirect Base bestandless.com.au
Domain IPs 13.210.38.93
Redirect IPs 13.35.238.111, 13.35.238.41, 13.35.238.75, 13.35.238.86, 2600:9000:2085:2e00:5:b93a:940:93a1, 2600:9000:2085:3200:5:b93a:940:93a1, 2600:9000:2085:3400:5:b93a:940:93a1, 2600:9000:2085:6e00:5:b93a:940:93a1, 2600:9000:2085:ac00:5:b93a:940:93a1, 2600:9000:2085:e00:5:b93a:940:93a1, 2600:9000:2085:e200:5:b93a:940:93a1, 2600:9000:2085:ec00:5:b93a:940:93a1
Response IP 13.35.238.111
Found Yes
Hash cc26e9da161b1876d96d9298cccc5753df48f67b6e72751c6deb06947447fc64
SimHash ac53579ecff8

Groups

*

Rule Path
Disallow /cart
Disallow /login
Disallow /checkout
Disallow /my-account
Disallow /search
Disallow /quickView
Disallow /Robots
Disallow /select
Disallow /LDBLU2021_1225402_BLACK_SOLID
Allow /

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.bestandless.com.au/sitemap.xml
Back to top

Comments

  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top