students.flinders.edu.au
robots.txt

Robots Exclusion Standard data for students.flinders.edu.au

Resource Scan

Scan Details

Site Domain students.flinders.edu.au
Base Domain flinders.edu.au
Scan Status Ok
Last Scan2025-03-03T15:05:27+00:00
Next Scan 2025-04-02T15:05:27+00:00

Last Scan

Scanned2025-03-03T15:05:27+00:00
URL https://students.flinders.edu.au/robots.txt
Domain IPs 13.35.202.10, 13.35.202.107, 13.35.202.109, 13.35.202.74, 2600:9000:2078:2600:2:bf79:1b40:93a1, 2600:9000:2078:6c00:2:bf79:1b40:93a1, 2600:9000:2078:8600:2:bf79:1b40:93a1, 2600:9000:2078:8a00:2:bf79:1b40:93a1, 2600:9000:2078:8e00:2:bf79:1b40:93a1, 2600:9000:2078:ac00:2:bf79:1b40:93a1, 2600:9000:2078:b000:2:bf79:1b40:93a1, 2600:9000:2078:c600:2:bf79:1b40:93a1
Response IP 13.35.202.74
Found Yes
Hash d04f6e7ff44bc1208b23ae7a8e704ec7af053285efcf79f928ab52210c101bf2
SimHash 6835fc56c730

Groups

discoverybot*

Rule Path
Disallow /

atomic_email_hunter*

Rule Path
Disallow /

sitebot*

Rule Path
Disallow /

larbin*

Rule Path
Disallow /

nutch*

Rule Path
Disallow /

libwww*

Rule Path
Disallow /

ahrefsbot*

Rule Path
Disallow /

java/1.7.0_04
java/1.6.6_26
java/1.6.6_26

Rule Path
Disallow /

cfnetwork*

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

*

Rule Path
Disallow /my-course/course-rules-2019
Disallow /my-course/course-rules-2020
Disallow /my-course/course-rules-2021

Warnings

  • 2 invalid lines.