abc-cdn.net.au
robots.txt

Robots Exclusion Standard data for abc-cdn.net.au

Resource Scan

Scan Details

Site Domain abc-cdn.net.au
Base Domain abc-cdn.net.au
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-06-22T10:08:52+00:00
Next Scan 2024-06-29T10:08:52+00:00

Last Successful Scan

Scanned2024-05-22T10:08:04+00:00
URL http://abc-cdn.net.au/robots.txt
Redirect https://www.abc.net.au/robots.txt
Redirect Domain www.abc.net.au
Redirect Base abc.net.au
Domain IPs 203.2.218.214
Redirect IPs 184.25.220.95
Response IP 23.36.252.109
Found Yes
Hash ab6550c09ac624ac02437d61ef2df6dc780f1a7b2e22eedb1b853b48f545ebb5
SimHash 4440b00e4f18

Groups

*

Rule Path
Disallow /classic/contact/concerts.htm
Disallow /classic/contact/default.htm
Disallow /classic/contact/eventsdiary.htm
Disallow /classic/contact/formerror.htm
Disallow /classic/contact/formthanks.htm
Disallow /classic/contact/general.htm
Disallow /classic/contact/limelight.htm
Disallow /classic/contact/mailinglist.htm
Disallow /classic/contact/music.htm
Disallow /classic/contact/presenter.htm
Disallow /classic/contact/website.htm
Disallow /classic/contact/word.htm
Disallow /xmlcontent/
Disallow /classicfm/
Disallow /iview/
Disallow /site-archive/
Disallow /corp/
Disallow /contact/
Disallow /homepage/2013/
Disallow /beta/
Disallow /abc4000/
Disallow /res/

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

flipboardproxy

Rule Path
Disallow /news/image/

Other Records

Field Value
crawl-delay 2

isec_bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

tineye-bot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Comments

  • robots.txt for https://www.abc.net.au/ -- ABC Online
  • OPSSD-340 2015/5/5
  • INNG-46: 2014-12-30
  • Added for corporate communications, as they have migrated to a new site
  • Added for Homepage Beta, prevent indexing during public beta
  • Added for WCMS Tennent testing, not a public