Log In Sign Up

applynow.net.au
robots.txt

Robots Exclusion Standard data for applynow.net.au

Resource Scan

Scan Details

Site Domain applynow.net.au
Base Domain applynow.net.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-23T23:52:42+00:00
Next Scan 2024-09-30T23:52:42+00:00

Last Successful Scan

Scanned2024-08-23T23:51:48+00:00
URL https://applynow.net.au/robots.txt
Domain IPs 108.157.254.50, 108.157.254.53, 108.157.254.54, 108.157.254.67
Response IP 108.157.254.50
Found Yes
Hash 68615296d8a418891663d7810f878f63d8d6a9b25de14e705bdec1645294a9db
SimHash 860d6d0d6640

Groups

*

Rule Path
Disallow /admin
Disallow /charts/
Disallow /images/
Disallow /stylesheets/
Disallow /javascript/
Disallow /w3c/
Disallow /404.html
Disallow /422.html
Disallow /500.html
Disallow /jobs/ni/
Disallow /Jobs/ni/
Disallow /applications/
Disallow /jobs/NSBC3-workforce-register

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Disallow: /
Back to top