cinch.org.nz
robots.txt

Robots Exclusion Standard data for cinch.org.nz

Archived Snapshots

Resource Scan

Scan Details

Site Domain	cinch.org.nz
Base Domain	cinch.org.nz
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-09-15T21:00:26+00:00
Next Scan	2024-12-14T21:00:26+00:00

Last Successful Scan

Scanned	2023-10-29T20:57:50+00:00
URL	https://cinch.org.nz/robots.txt
Redirect	https://www.cinch.org.nz:443/robots.txt
Redirect Domain	www.cinch.org.nz
Redirect Base	cinch.org.nz
Domain IPs	52.62.53.248, 52.64.125.164
Redirect IPs	52.65.28.196, 54.206.58.6
Response IP	54.206.58.6
Found	Yes
Hash	a238e1f3ef46167f75e2c41a997ea5792c0748f1b075b4c8127414bcddd1f7ca
SimHash	b029dbc3c1e7

Groups

*

Rule	Path
Disallow	/

Rule

Path

Disallow

googlebot

Rule	Path
Disallow	/categories/1
Disallow	/categories/2
Disallow	/categories/3
Disallow	/categories/4
Disallow	/categories/5
Disallow	/categories/6
Disallow	/categories/7
Disallow	/categories/8
Disallow	/categories/9
Disallow	/*?
Disallow	/keyword%3D*
Disallow	/filter%3D*
Disallow	*/entries/advanced-search

Rule

Path

Disallow

*/categories/1*

Disallow

*/categories/2*

Disallow

*/categories/3*

Disallow

*/categories/4*

Disallow

*/categories/5*

Disallow

*/categories/6*

Disallow

*/categories/7*

Disallow

*/categories/8*

Disallow

*/categories/9*

Disallow

/*?

Disallow

*/*keyword%3D*

Disallow

*/*filter%3D*

Disallow

*/entries/advanced-search

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

bingbot

Rule	Path
Disallow	/categories/1
Disallow	/categories/2
Disallow	/categories/3
Disallow	/categories/4
Disallow	/categories/5
Disallow	/categories/6
Disallow	/categories/7
Disallow	/categories/8
Disallow	/categories/9
Disallow	/*?
Disallow	/keyword%3D*
Disallow	/filter%3D*
Disallow	*/entries/advanced-search

Rule

Path

Disallow

*/categories/1*

Disallow

*/categories/2*

Disallow

*/categories/3*

Disallow

*/categories/4*

Disallow

*/categories/5*

Disallow

*/categories/6*

Disallow

*/categories/7*

Disallow

*/categories/8*

Disallow

*/categories/9*

Disallow

/*?

Disallow

*/*keyword%3D*

Disallow

*/*filter%3D*

Disallow

*/entries/advanced-search

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

msnbot

Rule	Path
Disallow	/categories/1
Disallow	/categories/2
Disallow	/categories/3
Disallow	/categories/4
Disallow	/categories/5
Disallow	/categories/6
Disallow	/categories/7
Disallow	/categories/8
Disallow	/categories/9
Disallow	/*?
Disallow	/keyword%3D*
Disallow	/filter%3D*
Disallow	*/entries/advanced-search

Rule

Path

Disallow

*/categories/1*

Disallow

*/categories/2*

Disallow

*/categories/3*

Disallow

*/categories/4*

Disallow

*/categories/5*

Disallow

*/categories/6*

Disallow

*/categories/7*

Disallow

*/categories/8*

Disallow

*/categories/9*

Disallow

/*?

Disallow

*/*keyword%3D*

Disallow

*/*filter%3D*

Disallow

*/entries/advanced-search

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

psbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yahoo-mmcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

nearsite

Rule	Path
Disallow	/

Rule

Path

Disallow

microsoft url control - 5.01.4511

Rule	Path
Disallow	/

Rule

Path

Disallow

website quester - www.esalesbiz.com/extra/

Rule	Path
Disallow	/

Rule

Path

Disallow

offline explorer/2.4

Rule	Path
Disallow	/

Rule

Path

Disallow

maxthon

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip/3.65 (http://www.spidersoft.com)

Rule	Path
Disallow	/

Rule

Path

Disallow

netzip-downloader/1.0.62 (win32; dec 7 1998)

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

Rule	Path
Disallow	/

Rule

Path

Disallow

zeus 97371 webster pro v2.9 win32

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver*

Rule	Path
Disallow	/

Rule

Path

Disallow

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

netzippy+(http://www.innerprise.net/usp-spider.asp)

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

searchpreview

Rule	Path
Disallow	/

Rule

Path

Disallow

realdownload

Rule	Path
Disallow	/

Rule

Path

Disallow

openfind

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin_2.6.2 larbin2.6.2@unspecified.mail

Rule	Path
Disallow	/

Rule

Path

Disallow

superbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ia_archiver

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack [nc,or]

Rule	Path
Disallow	/

Rule

Path

Disallow

miixpc/4.2

Rule	Path
Disallow	/

Rule

Path

Disallow

netants/1.25

Rule	Path
Disallow	/

Rule

Path

Disallow

website.quester

Rule	Path
Disallow	/

Rule

Path

Disallow

jetcar

Rule	Path
Disallow	/

Rule

Path

Disallow

webgo is

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

webauto

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

tocrawl/urldispatcher

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin_2.6.2 larbin@correa.org

Rule	Path
Disallow	/

Rule

Path

Disallow

urly warning

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin (samualt9@bigfoot.com)

Rule	Path
Disallow	/

Rule

Path

Disallow

pavuk

Rule	Path
Disallow	/

Rule

Path

Disallow

image stripper

Rule	Path
Disallow	/

Rule

Path

Disallow

teleport pro/1.29.1634

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

aipbot

Rule

Path

Disallow

internet ninja 4.0

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webauto/3.40 (win98; i)

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

black hole

Rule

Path

Disallow

webzip/4.0 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.21

Rule

Path

Disallow

url control

Rule

Path

Disallow

larbin_2.6.2 (vitalbox1@hotmail.com)

Rule

Path

Disallow

linknzbot*

Rule

Path

Disallow

webcopier v3.2a

Rule

Path

Disallow

zeus 82016 webster pro v2.9 win32

Rule

Path

Disallow

nutch

Rule

Path

Disallow

bullseye/1.0

Rule

Path

Disallow

webzip/4.21 (http://www.spidersoft.com)

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

disco pump 3.2

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

linkscan/8.1a unix disallow: /

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

realdownload/4.0.0.42

Rule

Path

Disallow

webzip/3.80 (http://www.spidersoft.com)

Rule

Path

Disallow

offline explorer/1.7

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

webstripper/2.19

Rule

Path

Disallow

openfind data gathere

Rule

Path

Disallow

mass downloader

Rule

Path

Disallow

java*

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

rma

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

webemailextrac.*

Rule

Path

Disallow

webzip/5.0 pr1 (http://www.spidersoft.com)

Rule

Path

Disallow

zeus 11652 webster pro v2.9 win32

Rule

Path

Disallow

netzip

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

smartdownload/1.2.77 (win32; feb 1 2000)

Rule

Path

Disallow

webzip/5.0 (http://www.spidersoft.com)

Rule

Path

Disallow

image sucker

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

turnitinbot/1.5

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

microsoft url control - 6.00.8169

Rule

Path

Disallow

linkscan/8.1a unix

Rule

Path

Disallow

microsoft url control*

Rule

Path

Disallow

missigua locator 1.9

Rule

Path

Disallow

mister pix ii 2.02a

Rule

Path

Disallow

httplib

Rule

Path

Disallow

webzip/4.1 (http://www.spidersoft.com)

Rule

Path

Disallow

bordermanager*

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

bilbo/2.3b-unix

Rule

Path

Disallow

moget

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

larbin_2.6.2 (listonatccdotgatechdotedu)

Rule

Path

Disallow

larbin_2.6.2 listonatccdotgatechdotedu

Rule

Path

Disallow

webreaper [webreaper@otway.com]

Rule

Path

Disallow

httptool*

Rule

Path

Disallow

moget/2.1

Rule

Path

Disallow

linknzbot

Rule

Path

Disallow

aqua_products

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

websauger 1.20k

Rule

Path

Disallow

websauger

Rule

Path

Disallow

webstripper/2.13

Rule

Path

Disallow

webster.pro

Rule

Path

Disallow

offline explorer/1.4

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

offline explorer/1.9

Rule

Path

Disallow

webcopier v2.7a

Rule

Path

Disallow

blowfish

Rule

Path

Disallow

zeus 30747 webster pro v2.9 win32

Rule

Path

Disallow

joc web spider

Rule

Path

Disallow

offline explorer/1.2

Rule

Path

Disallow

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

larbin_2.6.2 (kabura@sushi.com)

Rule

Path

Disallow

webcopier v3.0.1

Rule

Path

Disallow

libweb/clshttpdisallow: /

Rule

Path

Disallow

keyword.density

Rule

Path

Disallow

internet ninja 5.0

Rule

Path

Disallow

realdownload/4.0.0.40

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

copernic

Rule

Path

Disallow

web.image.collector

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

netzip downloader 1.0 win32(nov 12 1998)

Rule

Path

Disallow

webcollage/1.77

Rule

Path

Disallow

backdoorbot

Rule

Path

Disallow

harvest

Rule

Path

Disallow

mister.pix

Rule

Path

Disallow

sootle

Rule

Path

Disallow

grub

Rule

Path

Disallow

discofinder

Rule

Path

Disallow

copyrightcheck

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

website quester

Rule

Path

Disallow

http://www.almaden.ibm.com/cs/crawler

Rule

Path

Disallow

voideye

Rule

Path

Disallow

larbin samualt9@bigfoot.com

Rule

Path

Disallow

linknzbot 2004

Rule

Path

Disallow

ecatch/3.0

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

offline explorer/2.0

Rule

Path

Disallow

zeus 44238 webster pro v2.9 win32

Rule

Path

Disallow

microsoft url control - 6.01.9782

Rule

Path

Disallow

msrbot

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

nutch*

Rule

Path

Disallow

blowfish/1.0

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

webreaper [info@webreaper.net]

Rule

Path

Disallow

cherrypickerelite/1.0

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

netants

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

teleport pro

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

bunnyslippers

Rule

Path

Disallow

teleport pro/1.29.1847

Rule

Path

Disallow

net vampire

Rule

Path

Disallow

webstripper/2.16

Rule

Path

Disallow

webcopier v2.8

Rule

Path

Disallow

mister pix ii 2.01

Rule

Path

Disallow

offline explorer/1.6

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

reget

Rule

Path

Disallow

zeus 63567 webster pro v2.9 win32

Rule

Path

Disallow

webreaper v9.7 - www.webreaper.net

Rule

Path

Disallow

zeus 11389 webster pro v2.9 win32

Rule

Path

Disallow

mass downloader/2.2

Rule

Path

Disallow

webzip/2.75 (http://www.spidersoft.com)

Rule

Path

Disallow

hmview

Rule

Path

Disallow

larbin_2.6.2 kabura@sushi.com

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

webzip/5.0

Rule

Path

Disallow

openbot

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

zeus

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

mister pix version.dll

Rule

Path

Disallow

true_robot/1.0

Rule

Path

Disallow

aipbot*

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

larbin

Rule

Path

Disallow

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

nutchcvs/0.06-dev

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

netspider

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

superhttp/1.0

Rule

Path

Disallow

dumbot

Rule

Path

Disallow

realdownload/4.0.0.41

Rule

Path

Disallow

zeus 90872 webster pro v2.9 win32

Rule

Path

Disallow

websauger 1.20j

Rule

Path

Disallow

crescent internet toolpak httpole control v.1.0

Rule

Path

Disallow

midown tool

Rule

Path

Disallow

webcopier v2.5

Rule

Path

Disallow

download demon/3.2.0.8

Rule

Path

Disallow

fast-webcrawler/3.6/firstpage

Rule

Path

Disallow

website quester - www.asona.org

Rule

Path

Disallow

internet ninja 6.0

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

linkscan/8.1a.unix

Rule

Path

Disallow

net vampire/3.0

Rule

Path

Disallow

webreaper v9.1 - www.otway.com/webreaper

Rule

Path

Disallow

alexibot

Rule

Path

Disallow

websauger 1.20b

Rule

Path

Disallow

httrack

Rule

Path

Disallow

webreaper vwebreaper v7.3 - www,otway.com/webreaper

Rule

Path

Disallow

proximic

Rule

Path

Disallow

Comments

Disallow all in case there are ones not in our blacklist
Allow only important bots, but force bots to only use the a-z path to listings
blacklist

Warnings

42 invalid lines.

cinch.org.nzrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

googlebot

Other Records

bingbot

Other Records

msnbot

Other Records

blexbot

ahrefsbot

psbot

yahoo-mmcrawler

nearsite

microsoft url control - 5.01.4511

website quester - www.esalesbiz.com/extra/

offline explorer/2.4

maxthon

webzip/3.65 (http://www.spidersoft.com)

netzip-downloader/1.0.62 (win32; dec 7 1998)

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

zeus 97371 webster pro v2.9 win32

ia_archiver*

builtbottough

netzippy+(http://www.innerprise.net/usp-spider.asp)

webzip/4.0

searchpreview

realdownload

openfind

larbin_2.6.2 larbin2.6.2@unspecified.mail

superbot

ia_archiver

httrack [nc,or]

miixpc/4.2

netants/1.25

website.quester

jetcar

webgo is

aipbot/1.0

webauto

bullseye

tocrawl/urldispatcher

larbin_2.6.2 larbin@correa.org

urly warning

larbin (samualt9@bigfoot.com)

pavuk

image stripper

teleport pro/1.29.1634

extractorpro

botrighthere

infonavirobot

aipbot

internet ninja 4.0

webbandit

webauto/3.40 (win98; i)

superhttp

black hole

webzip/4.0 (http://www.spidersoft.com)

webzip/4.21

url control

larbin_2.6.2 (vitalbox1@hotmail.com)

linknzbot*

webcopier v3.2a

zeus 82016 webster pro v2.9 win32

nutch

bullseye/1.0

webzip/4.21 (http://www.spidersoft.com)

www-collector-e

disco pump 3.2

python-urllib

linkscan/8.1a unix disallow: /

spankbot

sitesnagger

realdownload/4.0.0.42

webzip/3.80 (http://www.spidersoft.com)

offline explorer/1.7

szukacz/1.4

cinch.org.nz
robots.txt