Log In Sign Up

rnz.co.nz
robots.txt

Robots Exclusion Standard data for rnz.co.nz

Resource Scan

Scan Details

Site Domain rnz.co.nz
Base Domain rnz.co.nz
Scan Status Ok
Last Scan2024-09-28T17:30:50+00:00
Next Scan 2024-10-28T17:30:50+00:00

Last Scan

Scanned2024-09-28T17:30:50+00:00
URL https://rnz.co.nz/robots.txt
Redirect https://www.rnz.co.nz/robots.txt
Redirect Domain www.rnz.co.nz
Redirect Base rnz.co.nz
Domain IPs 151.101.130.133, 151.101.194.133, 151.101.2.133, 151.101.66.133, 2a04:4e42:200::645, 2a04:4e42:400::645, 2a04:4e42:600::645, 2a04:4e42::645
Redirect IPs 151.101.130.133, 151.101.194.133, 151.101.2.133, 151.101.66.133
Response IP 199.232.46.133
Found Yes
Hash 1c42e3154254955a08d32aca20c096bda7c031ba45631276766d748ed3f46a43
SimHash 4c00bc35c3e1

Groups

*

Rule Path
Disallow /admin/

Other Records

Field Value
crawl-delay 7

psbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

bingbot
msnbot
yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

wget

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.rnz.co.nz/sitemap/sitemap.xml.gz
Back to top

Comments

  • Direct the most annoying crawlers not to index
  • Slow down the high-download / low-return crawlers
Back to top