thecoast.net.nz
robots.txt

Robots Exclusion Standard data for thecoast.net.nz

Archived Snapshots

Resource Scan

Scan Details

Site Domain	thecoast.net.nz
Base Domain	thecoast.net.nz
Scan Status	Ok
Last Scan	2025-08-25T14:19:42+00:00
Next Scan	2025-09-01T14:19:42+00:00

Last Scan

Scanned	2025-08-25T14:19:42+00:00
URL	https://thecoast.net.nz/robots.txt
Redirect	https://www.thecoast.net.nz/robots.txt
Redirect Domain	www.thecoast.net.nz
Redirect Base	thecoast.net.nz
Domain IPs	104.18.6.214, 104.18.7.214, 2606:4700::6812:6d6, 2606:4700::6812:7d6
Redirect IPs	104.18.6.214, 104.18.7.214, 2606:4700::6812:6d6, 2606:4700::6812:7d6
Response IP	104.18.6.214
Found	Yes
Hash	878e6ac4805310edff727e97439f528ef48018c4237c4a1889f32c73e48026a5
SimHash	322192f105e7

Groups

bingbot

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

msnbot

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

slurp

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

yahoo-newscrawler

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

googlebot

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

aasa-bot/1.0.0

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

aasa-bot

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

apple-bot/1.0.0

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

apple-bot

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

googlebot-news

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

echoboxbot/1.0

Rule	Path
Disallow	/search/

Rule

Path

Disallow

/search/

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

psbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yahoo-mmcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Allow	/

Rule

Path

Allow

aipbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot*

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

archiveteam

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bilbo/2.3b-unix

Rule	Path
Disallow	/

Rule

Path

Disallow

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bordermanager*

Rule	Path
Disallow	/

Rule

Path

Disallow

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

copernic

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent internet toolpak http ole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent internet toolpak httpole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

disco pump 3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

discofinder

Rule	Path
Disallow	/

Rule

Path

Disallow

download demon/3.2.0.8

Rule	Path
Disallow	/

Rule

Path

Disallow

dumbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ecatch/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

emailwolf

Rule	Path
Disallow	/

Rule

Path

Disallow

erocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

fast-webcrawler/3.6/firstpage

Rule

Path

Disallow

flashget

Rule

Path

Disallow

flashget webwasher 3.2

Rule

Path

Disallow

frontpage [nc,or]

Rule

Path

Disallow

getright/3.1

Rule

Path

Disallow

getright/3.3

Rule

Path

Disallow

getright/4.1.0

Rule

Path

Disallow

getright/4.2

Rule

Path

Disallow

getright/4.2b (portuguxeas)

Rule

Path

Disallow

getright/4.2c

Rule

Path

Disallow

getright/4.3

Rule

Path

Disallow

getright/4.5

Rule

Path

Disallow

getright/4.5a

Rule

Path

Disallow

getright/4.5b

Rule

Path

Disallow

getright/4.5b3

Rule

Path

Disallow

getright/4.5b6

Rule

Path

Disallow

getright/4.5b7

Rule

Path

Disallow

getright/4.5c

Rule

Path

Disallow

getright/4.5d

Rule

Path

Disallow

getright/4.5e

Rule

Path

Disallow

getright/5.0beta1

Rule

Path

Disallow

getright/5.0beta2

Rule

Path

Disallow

go!zilla (www.gozilla.com)

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

grub

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

harvest

Rule

Path

Disallow

hmview

Rule

Path

Disallow

http://www.almaden.ibm.com/cs/crawler

Rule

Path

Disallow

httplib

Rule

Path

Disallow

httptool*

Rule

Path

Disallow

httrack

Rule

Path

Disallow

httrack [nc,or]

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

image stripper

Rule

Path

Disallow

image sucker

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

internet ninja 4.0

Rule

Path

Disallow

internet ninja 5.0

Rule

Path

Disallow

internet ninja 6.0

Rule

Path

Disallow

java*

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

joc web spider

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword.density

Rule

Path

Disallow

larbin

Rule

Path

Disallow

larbin (samualt9@bigfoot.com)

Rule

Path

Disallow

larbin samualt9@bigfoot.com

Rule

Path

Disallow

larbin_2.6.2 (kabura@sushi.com)

Rule

Path

Disallow

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

Rule

Path

Disallow

larbin_2.6.2 (listonatccdotgatechdotedu)

Rule

Path

Disallow

larbin_2.6.2 (vitalbox1@hotmail.com)

Rule

Path

Disallow

larbin_2.6.2 kabura@sushi.com

Rule

Path

Disallow

larbin_2.6.2 larbin@correa.org

Rule

Path

Disallow

larbin_2.6.2 larbin2.6.2@unspecified.mail

Rule

Path

Disallow

larbin_2.6.2 listonatccdotgatechdotedu

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

libweb/clshttpdisallow: /

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

linknzbot

Rule

Path

Disallow

linknzbot 2004

Rule

Path

Disallow

linknzbot*

Rule

Path

Disallow

linkscan/8.1a unix

Rule

Path

Disallow

linkscan/8.1a unix disallow: /

Rule

Path

Disallow

linkscan/8.1a.unix

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

lwp

Rule

Path

Disallow

lwp*

Rule

Path

Disallow

lwp*

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwp-trivial/1.34

Rule

Path

Disallow

mass downloader

Rule

Path

Disallow

mass downloader/2.2

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

maxthon

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

microsoft url control - 5.01.4511

Rule

Path

Disallow

microsoft url control - 6.00.8169

Rule

Path

Disallow

microsoft url control - 6.01.9782

Rule

Path

Disallow

microsoft url control*

Rule

Path

Disallow

midown tool

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

miixpc/4.2

Rule

Path

Disallow

missigua locator 1.9

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

mister pix ii 2.01

Rule

Path

Disallow

mister pix ii 2.02a

Rule

Path

Disallow

mister pix version.dll

Rule

Path

Disallow

mister.pix

Rule

Path

Disallow

moget

Rule

Path

Disallow

moget/2.1

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

msrbot

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

net vampire

Rule

Path

Disallow

net vampire/3.0

Rule

Path

Disallow

netants

Rule

Path

Disallow

netants/1.25

Rule

Path

Disallow

netspider

Rule

Path

Disallow

netzip

Rule

Path

Disallow

netzip downloader 1.0 win32(nov 12 1998)

Rule

Path

Disallow

netzip-downloader/1.0.62 (win32; dec 7 1998)

Rule

Path

Disallow

netzippy+(http://www.innerprise.net/usp-spider.asp)

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

nutch

Rule

Path

Disallow

nutch*

Rule

Path

Disallow

nutchcvs/0.06-dev

Rule

Path

Disallow

offline explorer/1.2

Rule

Path

Disallow

offline explorer/1.4

Rule

Path

Disallow

offline explorer/1.6

Rule

Path

Disallow

offline explorer/1.7

Rule

Path

Disallow

offline explorer/1.9

Rule

Path

Disallow

offline explorer/2.0

Rule

Path

Disallow

offline explorer/2.4

Rule

Path

Disallow

openbot

Rule

Path

Disallow

openfind

Rule

Path

Disallow

openfind data gathere

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

realdownload/4.0.0.40

Rule

Path

Disallow

realdownload/4.0.0.41

Rule

Path

Disallow

realdownload/4.0.0.42

Rule

Path

Disallow

reget

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

rma

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

smartdownload/1.2.77 (win32; feb 1 2000)

Rule

Path

Disallow

sootle

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

superhttp/1.0

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

teleport pro

Rule

Path

Disallow

teleport pro/1.29.1634

Rule

Path

Disallow

teleport pro/1.29.1847

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

true_robot/1.0

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

turnitinbot/1.5

Rule

Path

Disallow

url control

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

voideye

Rule

Path

Disallow

wayback machine

Rule

Path

Disallow

web.image.collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webauto/3.40 (win98; i)

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

webcollage/1.77

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webcopier v2.5

Rule

Path

Disallow

webcopier v2.7a

Rule

Path

Disallow

webcopier v2.8

Rule

Path

Disallow

webcopier v3.0.1

Rule

Path

Disallow

webcopier v3.2a

Rule

Path

Disallow

webemailextrac.*

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webgo is

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

webreaper [info@webreaper.net]

Rule

Path

Disallow

webreaper [webreaper@otway.com]

Rule

Path

Disallow

webreaper v9.1 - www.otway.com/webreaper

Rule

Path

Disallow

webreaper v9.7 - www.webreaper.net

Rule

Path

Disallow

webreaper vwebreaper v7.3 - www,otway.com/webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

websauger 1.20b

Rule

Path

Disallow

websauger 1.20j

Rule

Path

Disallow

websauger 1.20k

Rule

Path

Disallow

website quester

Rule

Path

Disallow

website quester - www.asona.org

Rule

Path

Disallow

website quester - www.esalesbiz.com/extra/

Rule

Path

Disallow

website.quester

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

webster.pro

Rule

Path

Disallow

webstripper/2.13

Rule

Path

Disallow

webstripper/2.16

Rule

Path

Disallow

webstripper/2.19

Rule

Path

Disallow

webzip/2.75 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/3.65 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/3.80 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.0

Rule

Path

Disallow

webzip/4.0 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.1 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.21

Rule

Path

Disallow

webzip/4.21 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/5.0

Rule

Path

Disallow

webzip/5.0 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/5.0 pr1 (http://www.spidersoft.com)

Rule

Path

Disallow

wget

Rule

Path

Disallow

wget/1.5.2

Rule

Path

Disallow

wget/1.7

Rule

Path

Disallow

wget/1.8.2

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zeus 11389 webster pro v2.9 win32

Rule

Path

Disallow

zeus 11652 webster pro v2.9 win32

Rule

Path

Disallow

zeus 30747 webster pro v2.9 win32

Rule

Path

Disallow

zeus 44238 webster pro v2.9 win32

Rule

Path

Disallow

zeus 63567 webster pro v2.9 win32

Rule

Path

Disallow

zeus 82016 webster pro v2.9 win32

Rule

Path

Disallow

zeus 90872 webster pro v2.9 win32

Rule

Path

Disallow

zeus 97371 webster pro v2.9 win32

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.thecoast.net.nz/sitemap.xml

Comments

robots.txt for https://www.thecoast.net.nz
Good Bots allowed
Image Bots
Site scrapers & other known bad bots that are completely disallowed

Warnings

42 invalid lines.

thecoast.net.nzrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

bingbot

msnbot

slurp

Other Records

yahoo-newscrawler

Other Records

googlebot

aasa-bot/1.0.0

aasa-bot

apple-bot/1.0.0

apple-bot

googlebot-news

echoboxbot/1.0

Other Records

psbot

yahoo-mmcrawler

grapeshot

aipbot

aipbot*

aipbot/1.0

alexibot

aqua_products

archive.org_bot

archiveteam

backdoorbot

bilbo/2.3b-unix

black hole

blowfish

blowfish/1.0

bordermanager*

botrighthere

builtbottough

bullseye

bullseye/1.0

bunnyslippers

cherrypickerelite/1.0

copernic

copyrightcheck

crescent internet toolpak http ole control v.1.0

crescent internet toolpak httpole control v.1.0

disco pump 3.2

discofinder

download demon/3.2.0.8

dumbot

ecatch/3.0

emailwolf

erocrawler

extractorpro

fast-webcrawler/3.6/firstpage

flashget

flashget webwasher 3.2

frontpage [nc,or]

getright/3.1

getright/3.3

getright/4.1.0

getright/4.2

getright/4.2b (portuguxeas)

getright/4.2c

getright/4.3

getright/4.5

getright/4.5a

getright/4.5b

getright/4.5b3

getright/4.5b6

getright/4.5b7

getright/4.5c

getright/4.5d

getright/4.5e

getright/5.0beta1

getright/5.0beta2

go!zilla (www.gozilla.com)

go-ahead-got-it

grub

grub-client

harvest

thecoast.net.nz
robots.txt