Log In Sign Up

pennanthil-h.schools.nsw.gov.au
robots.txt

Robots Exclusion Standard data for pennanthil-h.schools.nsw.gov.au

Resource Scan

Scan Details

Site Domain pennanthil-h.schools.nsw.gov.au
Base Domain nsw.gov.au
Scan Status Ok
Last Scan2024-10-28T06:24:25+00:00
Next Scan 2024-11-27T06:24:25+00:00

Last Scan

Scanned2024-10-28T06:24:25+00:00
URL https://pennanthil-h.schools.nsw.gov.au/robots.txt
Domain IPs 18.65.244.126, 18.65.244.23, 18.65.244.57, 18.65.244.86
Response IP 18.65.244.23
Found Yes
Hash 0b78953e79c091d45d61c37fe172164fe27a01ee5074bcd0cd55259fc87564b4
SimHash 211df7b26013

Groups

*

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot-image

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

googlebot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot-media

Rule Path
Disallow /

bingbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

bingbot

Rule Path
Disallow /gallery
Disallow /newsletter

msnbot

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

msnbot

Rule Path
Disallow /gallery
Disallow /newsletter

slurp

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

slurp

Rule Path
Disallow /gallery
Disallow /newsletter

baiduspider-image

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

baiduspider

Rule Path
Disallow /gallery
Disallow /newsletter

yandex

Rule Path
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

yandex

Rule Path
Disallow /gallery
Disallow /newsletter

applebot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

duckduckbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /gallery
Disallow /newsletter
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.bmp$

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /
Back to top

Comments

  • For any user-agent not specifically included.
  • Google
  • Exclude images from Google Images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths. Googlebot does not use Crawl-delay.
  • Bing
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yahoo!
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Baidu
  • Exclude images.
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Yandex
  • Exclude all allowed image types.
  • Exclude specific paths, including all child paths.
  • Others
  • SemrushBot
Back to top