aarg.com.au
robots.txt

Robots Exclusion Standard data for aarg.com.au

Resource Scan

Scan Details

Site Domain aarg.com.au
Base Domain aarg.com.au
Scan Status Ok
Last Scan2025-05-04T21:22:56+00:00
Next Scan 2025-06-03T21:22:56+00:00

Last Scan

Scanned2025-05-04T21:22:56+00:00
URL https://aarg.com.au/robots.txt
Redirect https://www.aarg.com.au/robots.txt
Redirect Domain www.aarg.com.au
Redirect Base aarg.com.au
Domain IPs 199.34.228.47
Redirect IPs 199.34.228.47
Response IP 199.34.228.47
Found Yes
Hash 70fbb614fc76f7ed256bf9d7c72555ae8bc7593b4de21d30f5ac7d45e0f8ca8d
SimHash 8c02de5ee693

Groups

nerdybot

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /ajax/
Disallow /apps/
Disallow /kit-aircraft.html
Disallow /raaf1.html
Disallow /armstrong-siddeley.html
Disallow /about1.html
Disallow /general-aviation-aircraft.html
Disallow /engines1.html
Disallow /ran.html
Disallow /commonwealth-aircraft-corporation.html
Disallow /dap--gaf-aircraft.html
Disallow /heron.html
Disallow /memberfiles.html
Disallow /gliders1.html
Disallow /memberarchives.html
Disallow /pratt-and-whitney.html
Disallow /rolls-royce.html

Other Records

Field Value
sitemap https://www.aarg.com.au/sitemap.xml