Log In Sign Up

ropescross-p.schools.nsw.edu.au
robots.txt

Robots Exclusion Standard data for ropescross-p.schools.nsw.edu.au

Resource Scan

Scan Details

Site Domain ropescross-p.schools.nsw.edu.au
Base Domain ropescross-p.schools.nsw.edu.au
Scan Status Ok
Last Scan2024-06-04T01:51:16+00:00
Next Scan 2024-07-04T01:51:16+00:00

Last Scan

Scanned2024-06-04T01:51:16+00:00
URL http://www.ropescross-p.schools.nsw.edu.au/robots.txt
Redirect https://ropescross-p.schools.nsw.gov.au/robots.txt
Redirect Domain ropescross-p.schools.nsw.gov.au
Redirect Base nsw.gov.au
Domain IPs 153.107.134.76
Redirect IPs 108.158.20.28, 108.158.20.32, 108.158.20.68, 108.158.20.76
Response IP 108.158.20.28
Found Yes
Hash 0b78953e79c091d45d61c37fe172164fe27a01ee5074bcd0cd55259fc87564b4
SimHash 211df7b26013

Groups

*

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot-image

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot-media

Rule Path
Disallow /

bingbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

bingbot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

msnbot

Rule Path
Disallow /gallery
Disallow /newsletter

slurp

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

slurp

Rule Path
Disallow /gallery
Disallow /newsletter

baiduspider-image

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

baiduspider

Rule Path
Disallow /gallery
Disallow /newsletter

yandex

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

yandex

Rule Path
Disallow /gallery
Disallow /newsletter

applebot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

duckduckbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /
Back to top

Comments

  • For any user-agent not specifically included.
  • Google
  • Exclude images from Google Images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths. Googlebot does not use Crawl-delay.
  • Bing
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yahoo!
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Baidu
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yandex
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Others
  • SemrushBot
Back to top