Log In Sign Up

angusandrobertson.com.au
robots.txt

Robots Exclusion Standard data for angusandrobertson.com.au

Resource Scan

Scan Details

Site Domain angusandrobertson.com.au
Base Domain angusandrobertson.com.au
Scan Status Ok
Last Scan2024-09-02T23:02:06+00:00
Next Scan 2024-10-02T23:02:06+00:00

Last Scan

Scanned2024-09-02T23:02:06+00:00
URL https://angusandrobertson.com.au/robots.txt
Redirect https://www.angusrobertson.com.au/robots.txt
Redirect Domain www.angusrobertson.com.au
Redirect Base angusrobertson.com.au
Domain IPs 13.227.254.108, 13.227.254.112, 13.227.254.127, 13.227.254.13
Redirect IPs 2600:9000:2721:1600:e:823d:cb80:93a1, 2600:9000:2721:1c00:e:823d:cb80:93a1, 2600:9000:2721:3400:e:823d:cb80:93a1, 2600:9000:2721:4000:e:823d:cb80:93a1, 2600:9000:2721:6600:e:823d:cb80:93a1, 2600:9000:2721:7a00:e:823d:cb80:93a1, 2600:9000:2721:8400:e:823d:cb80:93a1, 2600:9000:2721:ec00:e:823d:cb80:93a1, 54.230.71.12, 54.230.71.80, 54.230.71.90, 54.230.71.99
Response IP 3.165.102.125
Found Yes
Hash 6d6879ff2d7ea72a2f73cf71066e239dac3eb8337ccff3ba3c84e870d12ee1db
SimHash 3e44dc9ef8fb

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account
Disallow /logout
Disallow /*q%3D*
Disallow /*gclid%3D*
Disallow /*source%3D*
Disallow /*utm_medium%3D*
Disallow /*utm_source%3D*
Disallow /*utm_campaign%3D*
Disallow /*attachment_id%3D*
Disallow /*clickid%3D*
Disallow /*src%3D*
Disallow /*sc_src%3D*
Disallow /*sc_customer%3D*
Disallow /*sc_lid%3D*
Disallow /*sc_llid%3D*
Disallow /*sc_uid%3D*
Disallow /*utm_content%3D*
Disallow /*irclickid%3D*
Disallow /*zsrc%3D*
Disallow /*voucherCode%3D*
Disallow /*msclkid%3D*
Disallow /*addedToWishlist%3D*
Disallow /*uiel%3D*
Disallow /*sort%3D*
Disallow /*token%3D*
Disallow /*orderNumber%3D*
Disallow /*fbclid%3D*

Other Records

Field Value
crawl-delay 10

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

ahrefsbot
semrushbot
dotbot
linkpadbot
spbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.angusrobertson.com.au/sitemap-index.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top