amf.org.au
robots.txt

Robots Exclusion Standard data for amf.org.au

Resource Scan

Scan Details

Site Domain amf.org.au
Base Domain amf.org.au
Scan Status Ok
Last Scan5/22/2025, 4:16:18 AM
Next Scan 5/29/2025, 4:16:18 AM

Last Scan

Scanned5/22/2025, 4:16:18 AM
URL https://amf.org.au/robots.txt
Redirect https://www.alannahandmadeline.org.au/robots.txt
Redirect Domain www.alannahandmadeline.org.au
Redirect Base alannahandmadeline.org.au
Domain IPs 104.26.8.195, 104.26.9.195, 172.67.72.121, 2606:4700:20::681a:8c3, 2606:4700:20::681a:9c3, 2606:4700:20::ac43:4879
Redirect IPs 104.21.23.139, 172.67.211.83, 2606:4700:3033::6815:178b, 2606:4700:3035::ac43:d353
Response IP 104.21.23.139
Found Yes
Hash 2f5be6dc4c5ed55e7e2f670af9fd8b3018c3ed93ac8d0864a1fbbd288c202bd1
SimHash a8284f264fd8

Groups

*

Rule Path
Disallow /uploads/main/cms-manual/
Disallow /uploads/main/Demo/
Disallow /uploads/main/For-Developers/
Disallow /imager/main/cms-manual/
Disallow /imager/main/Demo/
Disallow /imager/main/For-Developers/
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /dynamic
Disallow /docs

Comments

  • Allow robots access to everything
  • It's also highly recommended that any URLs blocked in robots.txt also have an X-Robots-Tag
  • header, otherwise they could still potentially make it into search results
  • See http://webmasters.stackexchange.com/questions/65184/will-the-already-indexed-urls-be-removed-if-i-use-a-robots-txt