Log In Sign Up

angusrobertson.com.au
robots.txt

Robots Exclusion Standard data for angusrobertson.com.au

Resource Scan

Scan Details

Site Domain angusrobertson.com.au
Base Domain angusrobertson.com.au
Scan Status Ok
Last Scan2024-05-18T05:47:25+00:00
Next Scan 2024-06-17T05:47:25+00:00

Last Scan

Scanned2024-05-18T05:47:25+00:00
URL https://angusrobertson.com.au/robots.txt
Redirect https://www.angusrobertson.com.au/robots.txt
Redirect Domain www.angusrobertson.com.au
Redirect Base angusrobertson.com.au
Domain IPs 13.33.30.10, 13.33.30.23, 13.33.30.45, 13.33.30.92, 2600:9000:229f:3200:e:823d:cb80:93a1, 2600:9000:229f:4800:e:823d:cb80:93a1, 2600:9000:229f:6000:e:823d:cb80:93a1, 2600:9000:229f:6c00:e:823d:cb80:93a1, 2600:9000:229f:7400:e:823d:cb80:93a1, 2600:9000:229f:a800:e:823d:cb80:93a1, 2600:9000:229f:aa00:e:823d:cb80:93a1, 2600:9000:229f:de00:e:823d:cb80:93a1
Redirect IPs 18.67.181.112, 18.67.181.129, 18.67.181.30, 18.67.181.99, 2600:9000:229f:2e00:e:823d:cb80:93a1, 2600:9000:229f:600:e:823d:cb80:93a1, 2600:9000:229f:9200:e:823d:cb80:93a1, 2600:9000:229f:a400:e:823d:cb80:93a1, 2600:9000:229f:b200:e:823d:cb80:93a1, 2600:9000:229f:e00:e:823d:cb80:93a1, 2600:9000:229f:ee00:e:823d:cb80:93a1, 2600:9000:229f:f400:e:823d:cb80:93a1
Response IP 13.33.30.23
Found Yes
Hash 6d6879ff2d7ea72a2f73cf71066e239dac3eb8337ccff3ba3c84e870d12ee1db
SimHash 3e44dc9ef8fb

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account
Disallow /logout
Disallow /*q%3D*
Disallow /*gclid%3D*
Disallow /*source%3D*
Disallow /*utm_medium%3D*
Disallow /*utm_source%3D*
Disallow /*utm_campaign%3D*
Disallow /*attachment_id%3D*
Disallow /*clickid%3D*
Disallow /*src%3D*
Disallow /*sc_src%3D*
Disallow /*sc_customer%3D*
Disallow /*sc_lid%3D*
Disallow /*sc_llid%3D*
Disallow /*sc_uid%3D*
Disallow /*utm_content%3D*
Disallow /*irclickid%3D*
Disallow /*zsrc%3D*
Disallow /*voucherCode%3D*
Disallow /*msclkid%3D*
Disallow /*addedToWishlist%3D*
Disallow /*uiel%3D*
Disallow /*sort%3D*
Disallow /*token%3D*
Disallow /*orderNumber%3D*
Disallow /*fbclid%3D*

Other Records

Field Value
crawl-delay 10

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

ahrefsbot
semrushbot
dotbot
linkpadbot
spbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.angusrobertson.com.au/sitemap-index.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top