Log In Sign Up

beverlyhg-h.schools.nsw.gov.au
robots.txt

Robots Exclusion Standard data for beverlyhg-h.schools.nsw.gov.au

Resource Scan

Scan Details

Site Domain beverlyhg-h.schools.nsw.gov.au
Base Domain nsw.gov.au
Scan Status Ok
Last Scan2024-06-17T03:41:53+00:00
Next Scan 2024-07-17T03:41:53+00:00

Last Scan

Scanned2024-06-17T03:41:53+00:00
URL https://beverlyhg-h.schools.nsw.gov.au/robots.txt
Domain IPs 108.158.32.115, 108.158.32.129, 108.158.32.33, 108.158.32.75
Response IP 108.158.32.75
Found Yes
Hash 0b78953e79c091d45d61c37fe172164fe27a01ee5074bcd0cd55259fc87564b4
SimHash 211df7b26013

Groups

*

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot-image

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot-media

Rule Path
Disallow /

bingbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

bingbot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

msnbot

Rule Path
Disallow /gallery
Disallow /newsletter

slurp

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

slurp

Rule Path
Disallow /gallery
Disallow /newsletter

baiduspider-image

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

baiduspider

Rule Path
Disallow /gallery
Disallow /newsletter

yandex

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

yandex

Rule Path
Disallow /gallery
Disallow /newsletter

applebot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

duckduckbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /
Back to top

Comments

  • For any user-agent not specifically included.
  • Google
  • Exclude images from Google Images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths. Googlebot does not use Crawl-delay.
  • Bing
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yahoo!
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Baidu
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yandex
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Others
  • SemrushBot
Back to top