undertheradar.co.nz
robots.txt

Robots Exclusion Standard data for undertheradar.co.nz

Archived Snapshots

Resource Scan

Scan Details

Site Domain	undertheradar.co.nz
Base Domain	undertheradar.co.nz
Scan Status	Ok
Last Scan	2024-10-21T03:20:07+00:00
Next Scan	2024-11-20T03:20:07+00:00

Last Scan

Scanned	2024-10-21T03:20:07+00:00
URL	https://undertheradar.co.nz/robots.txt
Redirect	https://www.undertheradar.co.nz/robots.txt
Redirect Domain	www.undertheradar.co.nz
Redirect Base	undertheradar.co.nz
Domain IPs	119.47.121.241
Redirect IPs	119.47.121.241
Response IP	119.47.121.241
Found	Yes
Hash	0c0544a053ba7e3d5649cdcb048fd6a912ba8a0891b1983d28c1aa2116516042
SimHash	186d9341c1a7

Groups

*

Rule	Path
Disallow	/backstage/
Disallow	/cgi-bin/
Disallow	/assets/
Disallow	/audio/
Disallow	/mp3/
Disallow	/panels/
Disallow	/top5/
Disallow	/transfer/
Disallow	/flash/
Disallow	/images
Disallow	/images/artists/
Disallow	/images/items/
Disallow	/images/itemsthumbs/
Disallow	/images/labels/
Disallow	/images/labelsthumbs/
Disallow	/images/live/
Disallow	/images/newsImagesThumbs/
Disallow	/images/site
Disallow	/images/thumbs/
Disallow	/images/imagecache
Disallow	/images/imagecacheEx
Disallow	/images/imagecacheEx2
Disallow	/game/
Disallow	/gallery/
Disallow	/javascript/
Disallow	/doubleclick/

Rule

Path

Disallow

/backstage/

Disallow

/cgi-bin/

Disallow

/assets/

Disallow

/audio/

Disallow

/mp3/

Disallow

/panels/

Disallow

/top5/

Disallow

/transfer/

Disallow

/flash/

Disallow

/images

Disallow

/images/artists/

Disallow

/images/items/

Disallow

/images/itemsthumbs/

Disallow

/images/labels/

Disallow

/images/labelsthumbs/

Disallow

/images/live/

Disallow

/images/newsImagesThumbs/

Disallow

/images/site

Disallow

/images/thumbs/

Disallow

/images/imagecache

Disallow

/images/imagecacheEx

Disallow

/images/imagecacheEx2

Disallow

/game/

Disallow

/gallery/

Disallow

/javascript/

Disallow

/doubleclick/

iisbot

Rule	Path
Allow	/

Rule

Path

Allow

twitterbot

Rule	Path
Disallow	*
Allow	/images/*

Rule

Path

Disallow

Allow

/images/*

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot*

Rule	Path
Disallow	/

Rule

Path

Disallow

aipbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bilbo/2.3b-unix

Rule	Path
Disallow	/

Rule

Path

Disallow

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bordermanager*

Rule	Path
Disallow	/

Rule

Path

Disallow

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

copernic

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent internet toolpak http ole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent internet toolpak httpole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

disco pump 3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

discofinder

Rule	Path
Disallow	/

Rule

Path

Disallow

download demon/3.2.0.8

Rule	Path
Disallow	/

Rule

Path

Disallow

dumbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ecatch/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

emailwolf

Rule	Path
Disallow	/

Rule

Path

Disallow

erocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

fast-webcrawler/3.6/firstpage

Rule	Path
Disallow	/

Rule

Path

Disallow

flashget

Rule	Path
Disallow	/

Rule

Path

Disallow

flashget webwasher 3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

frontpage [nc,or]

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.1

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.3

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.2

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.2b (portuguxeas)

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.2c

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.3

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.5

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.5a

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.5b

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.5b3

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/4.5b6

Rule

Path

Disallow

getright/4.5b7

Rule

Path

Disallow

getright/4.5c

Rule

Path

Disallow

getright/4.5d

Rule

Path

Disallow

getright/4.5e

Rule

Path

Disallow

getright/5.0beta1

Rule

Path

Disallow

getright/5.0beta2

Rule

Path

Disallow

go!zilla (www.gozilla.com)

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

grub

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

harvest

Rule

Path

Disallow

hmview

Rule

Path

Disallow

http://www.almaden.ibm.com/cs/crawler

Rule

Path

Disallow

httplib

Rule

Path

Disallow

httptool*

Rule

Path

Disallow

httrack

Rule

Path

Disallow

httrack [nc,or]

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

ia_archiver*

Rule

Path

Disallow

image stripper

Rule

Path

Disallow

image sucker

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

internet ninja 4.0

Rule

Path

Disallow

internet ninja 5.0

Rule

Path

Disallow

internet ninja 6.0

Rule

Path

Disallow

java*

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

joc web spider

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword.density

Rule

Path

Disallow

larbin

Rule

Path

Disallow

larbin (samualt9@bigfoot.com)

Rule

Path

Disallow

larbin samualt9@bigfoot.com

Rule

Path

Disallow

larbin_2.6.2 (kabura@sushi.com)

Rule

Path

Disallow

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

Rule

Path

Disallow

larbin_2.6.2 (listonatccdotgatechdotedu)

Rule

Path

Disallow

larbin_2.6.2 (vitalbox1@hotmail.com)

Rule

Path

Disallow

larbin_2.6.2 kabura@sushi.com

Rule

Path

Disallow

larbin_2.6.2 larbin@correa.org

Rule

Path

Disallow

larbin_2.6.2 larbin2.6.2@unspecified.mail

Rule

Path

Disallow

larbin_2.6.2 listonatccdotgatechdotedu

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

libweb/clshttpdisallow: /

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

linknzbot

Rule

Path

Disallow

linknzbot 2004

Rule

Path

Disallow

linknzbot*

Rule

Path

Disallow

linkscan/8.1a unix

Rule

Path

Disallow

linkscan/8.1a unix disallow: /

Rule

Path

Disallow

linkscan/8.1a.unix

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

lwp

Rule

Path

Disallow

lwp*

Rule

Path

Disallow

lwp*

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwp-trivial/1.34

Rule

Path

Disallow

mass downloader

Rule

Path

Disallow

mass downloader/2.2

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

maxthon

Rule

Path

Disallow

microsoft url control

Rule

Path

Disallow

microsoft url control - 5.01.4511

Rule

Path

Disallow

microsoft url control - 6.00.8169

Rule

Path

Disallow

microsoft url control - 6.01.9782

Rule

Path

Disallow

microsoft url control*

Rule

Path

Disallow

midown tool

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

miixpc/4.2

Rule

Path

Disallow

missigua locator 1.9

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

mister pix ii 2.01

Rule

Path

Disallow

mister pix ii 2.02a

Rule

Path

Disallow

mister pix version.dll

Rule

Path

Disallow

mister.pix

Rule

Path

Disallow

moget

Rule

Path

Disallow

moget/2.1

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

msrbot

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

net vampire

Rule

Path

Disallow

net vampire/3.0

Rule

Path

Disallow

netants

Rule

Path

Disallow

netants/1.25

Rule

Path

Disallow

netspider

Rule

Path

Disallow

netzip

Rule

Path

Disallow

netzip downloader 1.0 win32(nov 12 1998)

Rule

Path

Disallow

netzip-downloader/1.0.62 (win32; dec 7 1998)

Rule

Path

Disallow

netzippy+(http://www.innerprise.net/usp-spider.asp)

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

nutch

Rule

Path

Disallow

nutch*

Rule

Path

Disallow

nutchcvs/0.06-dev

Rule

Path

Disallow

offline explorer/1.2

Rule

Path

Disallow

offline explorer/1.4

Rule

Path

Disallow

offline explorer/1.6

Rule

Path

Disallow

offline explorer/1.7

Rule

Path

Disallow

offline explorer/1.9

Rule

Path

Disallow

offline explorer/2.0

Rule

Path

Disallow

offline explorer/2.4

Rule

Path

Disallow

openbot

Rule

Path

Disallow

openfind

Rule

Path

Disallow

openfind data gathere

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

realdownload/4.0.0.40

Rule

Path

Disallow

realdownload/4.0.0.41

Rule

Path

Disallow

realdownload/4.0.0.42

Rule

Path

Disallow

reget

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

rma

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

smartdownload/1.2.77 (win32; feb 1 2000)

Rule

Path

Disallow

sootle

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

superhttp/1.0

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

teleport pro

Rule

Path

Disallow

teleport pro/1.29.1634

Rule

Path

Disallow

teleport pro/1.29.1847

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

true_robot/1.0

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

turnitinbot/1.5

Rule

Path

Disallow

url control

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

voideye

Rule

Path

Disallow

web.image.collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webauto/3.40 (win98; i)

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

webcollage/1.77

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webcopier v2.5

Rule

Path

Disallow

webcopier v2.7a

Rule

Path

Disallow

webcopier v2.8

Rule

Path

Disallow

webcopier v3.0.1

Rule

Path

Disallow

webcopier v3.2a

Rule

Path

Disallow

webemailextrac.*

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webgo is

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

webreaper [info@webreaper.net]

Rule

Path

Disallow

webreaper [webreaper@otway.com]

Rule

Path

Disallow

webreaper v9.1 - www.otway.com/webreaper

Rule

Path

Disallow

webreaper v9.7 - www.webreaper.net

Rule

Path

Disallow

webreaper vwebreaper v7.3 - www,otway.com/webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

websauger 1.20b

Rule

Path

Disallow

websauger 1.20j

Rule

Path

Disallow

websauger 1.20k

Rule

Path

Disallow

website quester

Rule

Path

Disallow

website quester - www.asona.org

Rule

Path

Disallow

website quester - www.esalesbiz.com/extra/

Rule

Path

Disallow

website.quester

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

webster.pro

Rule

Path

Disallow

webstripper/2.13

Rule

Path

Disallow

webstripper/2.16

Rule

Path

Disallow

webstripper/2.19

Rule

Path

Disallow

webzip/2.75 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/3.65 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/3.80 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.0

Rule

Path

Disallow

webzip/4.0 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.1 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/4.21

Rule

Path

Disallow

webzip/4.21 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/5.0

Rule

Path

Disallow

webzip/5.0 (http://www.spidersoft.com)

Rule

Path

Disallow

webzip/5.0 pr1 (http://www.spidersoft.com)

Rule

Path

Disallow

wget

Rule

Path

Disallow

wget/1.5.2

Rule

Path

Disallow

wget/1.7

Rule

Path

Disallow

wget/1.8.2

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zeus 11389 webster pro v2.9 win32

Rule

Path

Disallow

zeus 11652 webster pro v2.9 win32

Rule

Path

Disallow

zeus 30747 webster pro v2.9 win32

Rule

Path

Disallow

zeus 44238 webster pro v2.9 win32

Rule

Path

Disallow

zeus 63567 webster pro v2.9 win32

Rule

Path

Disallow

zeus 82016 webster pro v2.9 win32

Rule

Path

Disallow

zeus 90872 webster pro v2.9 win32

Rule

Path

Disallow

zeus 97371 webster pro v2.9 win32

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

googlebot

Rule

Path

Disallow

/js/

Allow

/js/classie.js

Allow

/js/search.js

Allow

/images/showImages/

Allow

/images/newsImages/

Allow

/panels/shows/showMoreRelatedAjax.php

Allow

/panels/shows/showsPopularTabsAjax.php

Allow

/panels/shows/showPanelListAjax.php

Allow

/panels/shows/venueMorePanelUpcoming.php

Allow

/js/jquery3.3.1.js

Allow

/js/galleria/

Allow

/js/movedom.js

Other Records

Field

Value

sitemap

https://www.undertheradar.co.nz/sitemap_index.php

Comments

Site scrapers & other known bad bots that are completely disallowed

Warnings

45 invalid lines.

undertheradar.co.nzrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

iisbot

twitterbot

amazonbot

aipbot

aipbot*

aipbot/1.0

alexibot

aqua_products

backdoorbot

bilbo/2.3b-unix

black hole

blowfish

blowfish/1.0

bordermanager*

botrighthere

builtbottough

bullseye

bullseye/1.0

bunnyslippers

cherrypickerelite/1.0

copernic

copyrightcheck

crescent internet toolpak http ole control v.1.0

crescent internet toolpak httpole control v.1.0

disco pump 3.2

discofinder

download demon/3.2.0.8

dumbot

ecatch/3.0

emailwolf

erocrawler

extractorpro

fast-webcrawler/3.6/firstpage

flashget

flashget webwasher 3.2

frontpage [nc,or]

getright/3.1

getright/3.3

getright/4.1.0

getright/4.2

getright/4.2b (portuguxeas)

getright/4.2c

getright/4.3

getright/4.5

getright/4.5a

getright/4.5b

getright/4.5b3

getright/4.5b6

getright/4.5b7

getright/4.5c

getright/4.5d

getright/4.5e

getright/5.0beta1

getright/5.0beta2

go!zilla (www.gozilla.com)

go-ahead-got-it

grub

grub-client

harvest

hmview

http://www.almaden.ibm.com/cs/crawler

httplib

httptool*

httrack

httrack [nc,or]

ia_archiver

ia_archiver*

image stripper

image sucker

infonavirobot

internet ninja 4.0

internet ninja 5.0

internet ninja 6.0

java*

undertheradar.co.nz
robots.txt