rust.co.uk robots.txt

*

Rule	Path
Disallow	/wp-admin/
Disallow	/wp-includes/
Disallow	/cgi-bin/
Disallow	/trackback/
Disallow	/comments/
Disallow	/xmlrpc.php
Disallow	/?s=
Allow	/wp-admin/admin-ajax.php

Rule

Path

Disallow

/wp-admin/

Disallow

/wp-includes/

Disallow

/cgi-bin/

Disallow

/trackback/

Disallow

/comments/

Disallow

/xmlrpc.php

Disallow

/?s=

Allow

/wp-admin/admin-ajax.php

a6-indexer

Rule	Path
Disallow	/

Rule

Path

Disallow

/

abonti

Rule	Path
Disallow	/

Rule

Path

Disallow

/

adidxbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

adscanner

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

ahrefsbot/6.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aitcsrobot/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexa

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexa toolbar

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alphabot/3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

/

alphaseobot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

amznkassocbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

apache-httpclient/4.5.2 (java/1.8.0_161)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

apache-httpclient/4.5.9 (java/1.8.0_222)

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

/

arachnophilia

Rule	Path
Disallow	/

Rule

Path

Disallow

/

archive.org_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aspider/0.09

Rule	Path
Disallow	/

Rule

Path

Disallow

/

aspiegelbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

asterias

Rule	Path
Disallow	/

Rule

Path

Disallow

/

auresys/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

awariosmartbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

b2w/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

/

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backdoorbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

/

backrub/.

Rule	Path
Disallow	/

Rule

Path

Disallow

/

big brother

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bizbot003

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bizbot04 kirk.overleaf.com

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blackboard safeassign

Rule	Path
Disallow	/

Rule

Path

Disallow

/

black.hole

Rule	Path
Disallow	/

Rule

Path

Disallow

/

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blackwidow

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blexbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blowfish

Rule	Path
Disallow	/

Rule

Path

Disallow

/

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bookmark search tool

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bot mailto:craftbot@yahoo.com

Rule	Path
Disallow	/

Rule

Path

Disallow

/

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

/

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bspider/1.0 libwww-perl/0.40

Rule	Path
Disallow	/

Rule

Path

Disallow

/

bubing

Rule

Path

Disallow

/

builtbottough

Rule

Path

Disallow

/

bullseye

Rule

Path

Disallow

/

bullseye/1.0

Rule

Path

Disallow

/

bunnyslippers

Rule

Path

Disallow

/

cactvs chemistry spider

Rule

Path

Disallow

/

careerbot

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

cegbfeieh

Rule

Path

Disallow

/

changedetection

Rule

Path

Disallow

/

checkbot/x.xx lwp/5.x

Rule

Path

Disallow

/

cheesebot

Rule

Path

Disallow

/

cherrypicker

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule

Path

Disallow

/

chinaclaw

Rule

Path

Disallow

/

cliqzbot

Rule

Path

Disallow

/

cloudservermarketspider

Rule

Path

Disallow

/

coccoc

Rule

Path

Disallow

/

coccocbot-image/1.0

Rule

Path

Disallow

/

coibotparser/2.0

Rule

Path

Disallow

/

combine/0.0

Rule

Path

Disallow

/

compspybot

Rule

Path

Disallow

/

conceptbot/0.3

Rule

Path

Disallow

/

copernic

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

crawler4j

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

/

custo

Rule

Path

Disallow

/

cyberpatrol sitecat webbot

Rule

Path

Disallow

/

cyberspyder/2.1

Rule

Path

Disallow

/

dataprovider

Rule

Path

Disallow

/

daumoa

Rule

Path

Disallow

/

deweb/1.01

Rule

Path

Disallow

/

disco

Rule

Path

Disallow

/

disco pump 3.0

Rule

Path

Disallow

/

disco pump 3.2

Rule

Path

Disallow

/

discofinder

Rule

Path

Disallow

/

dittospyder

Rule

Path

Disallow

/

domaincrawler

Rule

Path

Disallow

/

domain re-animator bot

Rule

Path

Disallow

/

domaintuno

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

download demon

Rule

Path

Disallow

/

download demon/3.2.0.8

Rule

Path

Disallow

/

download demon/3.5.0.11

Rule

Path

Disallow

/

dubaiindex

Rule

Path

Disallow

/

dumbot

Rule

Path

Disallow

/

ecatch

Rule

Path

Disallow

/

ecatch/3.0

Rule

Path

Disallow

/

eirgrabber

Rule

Path

Disallow

/

emailcollector

Rule

Path

Disallow

/

emailsiphon

Rule

Path

Disallow

/

emailwolf

Rule

Path

Disallow

/

enigmabot

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

es

Rule

Path

Disallow

/

e.ventures investment crawler

Rule

Path

Disallow

/

exabot

Rule

Path

Disallow

/

explorersearch

Rule

Path

Disallow

/

express webpictures

Rule

Path

Disallow

/

express webpictures (www.express-soft.com)

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

eyenetie

Rule

Path

Disallow

/

fairad client

Rule

Path

Disallow

/

felixide/1.0

Rule

Path

Disallow

/

fido/0.9 harvest/1.4.pl2

Rule

Path

Disallow

/

fish-search-robot

Rule

Path

Disallow

/

flashget

Rule

Path

Disallow

/

flashget webwasher 3.2

Rule

Path

Disallow

/

flaming attackbot

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

fr-crawler

Rule

Path

Disallow

/

freecrawl

Rule

Path

Disallow

/

frontpage

Rule

Path

Disallow

/

frontpage [nc,or]

Rule

Path

Disallow

/

funwebproducts

Rule

Path

Disallow

/

gaisbot

Rule

Path

Disallow

/

garlikcrawler

Rule

Path

Disallow

/

garlikcrawler/1.2

Rule

Path

Disallow

/

gather analyze provide

Rule

Path

Disallow

/

gcreep/1.0

Rule

Path

Disallow

/

getright

Rule

Path

Disallow

/

getright/2.11

Rule

Path

Disallow

/

getright/3.1

Rule

Path

Disallow

/

getright/3.2

Rule

Path

Disallow

/

getright/3.3

Rule

Path

Disallow

/

getright/3.3.3

Rule

Path

Disallow

/

getright/3.3.4

Rule

Path

Disallow

/

getright/4.0.0

Rule

Path

Disallow

/

getright/4.1.0

Rule

Path

Disallow

/

getright/4.1.1

Rule

Path

Disallow

/

getright/4.1.2

Rule

Path

Disallow

/

getright/4.2

Rule

Path

Disallow

/

getright/4.2b (portuguxeas)

Rule

Path

Disallow

/

getright/4.2c

Rule

Path

Disallow

/

getright/4.3

Rule

Path

Disallow

/

getright/4.5

Rule

Path

Disallow

/

getright/4.5a

Rule

Path

Disallow

/

getright/4.5b

Rule

Path

Disallow

/

getright/4.5b1

Rule

Path

Disallow

/

getright/4.5b2

Rule

Path

Disallow

/

getright/4.5b3

Rule

Path

Disallow

/

getright/4.5b6

Rule

Path

Disallow

/

getright/4.5b7

Rule

Path

Disallow

/

getright/4.5c

Rule

Path

Disallow

/

getright/4.5d

Rule

Path

Disallow

/

getright/4.5e

Rule

Path

Disallow

/

getright/5.0beta1

Rule

Path

Disallow

/

getright/5.0beta2

Rule

Path

Disallow

/

geturl.rexx v1.05

Rule

Path

Disallow

/

getweb!

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

gluten free crawler/1.0

Rule

Path

Disallow

/

go!zilla

Rule

Path

Disallow

/

go!zilla (www.gozilla.com)

Rule

Path

Disallow

/

go!zilla 3.3 (www.gozilla.com)

Rule

Path

Disallow

/

go!zilla 3.5 (www.gozilla.com)

Rule

Path

Disallow

/

go-ahead-got-it

Rule

Path

Disallow

/

golem/1.1

Rule

Path

Disallow

/

gosign-security-crawler

Rule

Path

Disallow

/

grabnet

Rule

Path

Disallow

/

grafula

Rule

Path

Disallow

/

grapeshot

Rule

Path

Disallow

/

grapeshotcrawler

Rule

Path

Disallow

/

gromit/1.0

Rule

Path

Disallow

/

grub

Rule

Path

Disallow

/

haosouspider

Rule

Path

Disallow

/

happyfunbot

Rule

Path

Disallow

/

harvest

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

hatena antenna

Rule

Path

Disallow

/

hazel's ferret web hopper

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

hmview

Rule

Path

Disallow

/

http banner detection

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

httrack

Rule

Path

Disallow

/

httrack 3.0

Rule

Path

Disallow

/

httrack [nc,or]

Rule

Path

Disallow

/

huaweisymantecspider

Rule

Path

Disallow

/

hubspot links crawler 2.0

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

hypestat/1.0

Rule

Path

Disallow

/

hã¢â¬â°mã¢â¬â°hã¢â¬â°kki/0.2

Rule

Path

Disallow

/

ia_archiver

Rule

Path

Disallow

/

iccrawler

Rule

Path

Disallow

/

icjobs

Rule

Path

Disallow

/

image stripper

Rule

Path

Disallow

/

image sucker

Rule

Path

Disallow

/

implisensebot

Rule

Path

Disallow

/

inagist.com url crawler

Rule

Path

Disallow

/

incywincy/1.0b1

Rule

Path

Disallow

/

indeedbot

Rule

Path

Disallow

/

indy library

Rule

Path

Disallow

/

indy library [nc,or]

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

informant

Rule

Path

Disallow

/

ingrid/0.1

Rule

Path

Disallow

/

interget

Rule

Path

Disallow

/

internet ninja

Rule

Path

Disallow

/

internet ninja 4.0

Rule

Path

Disallow

/

internet ninja 5.0

Rule

Path

Disallow

/

internet ninja 6.0

Rule

Path

Disallow

/

internet-structure-research-project-bot

Rule

Path

Disallow

/

iodc; odysseus

Rule

Path

Disallow

/

ips-agent

Rule

Path

Disallow

/

iron33/1.0.2

Rule

Path

Disallow

/

israelisearch/1.0

Rule

Path

Disallow

/

istellabot

Rule

Path

Disallow

/

it2media-domain-crawler

Rule

Path

Disallow

/

iti spider

Rule

Path

Disallow

/

jamesbot

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

jetcar

Rule

Path

Disallow

/

jobboersebot

Rule

Path

Disallow

/

jobs.de-robot

Rule

Path

Disallow

/

joc web spider

Rule

Path

Disallow

/

jubiirobot

Rule

Path

Disallow

/

jikespider

Rule

Path

Disallow

/

jumpstation

Rule

Path

Disallow

/

katipo/1.0

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

kenjin.spider

Rule

Path

Disallow

/

keyword density/0.9

Rule

Path

Disallow

/

keyword.density

Rule

Path

Disallow

/

kit-fireball/2.0 libwww/5.0a

Rule

Path

Disallow

/

kraken

Rule

Path

Disallow

/

labelgrab/1.1

Rule

Path

Disallow

/

larbin

Rule

Path

Disallow

/

larbin (samualt9@bigfoot.com)

Rule

Path

Disallow

/

larbin samualt9@bigfoot.com

Rule

Path

Disallow

/

larbin_2.6.2 (kabura@sushi.com)

Rule

Path

Disallow

/

larbin_2.6.2 (larbin2.6.2@unspecified.mail)

Rule

Path

Disallow

/

larbin_2.6.2 (listonatccdotgatechdotedu)

Rule

Path

Disallow

/

larbin_2.6.2 (vitalbox1@hotmail.com)

Rule

Path

Disallow

/

larbin_2.6.2 kabura@sushi.com

Rule

Path

Disallow

/

larbin_2.6.2 larbin2.6.2@unspecified.mail

Rule

Path

Disallow

/

larbin_2.6.2 larbin@correa.org

Rule

Path

Disallow

/

larbin_2.6.2 listonatccdotgatechdotedu

Rule

Path

Disallow

/

larbin_2.6.2 vitalbox1@hotmail.com

Rule

Path

Disallow

/

leechftp

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

liebaofast

Rule

Path

Disallow

/

lightspeedsystemscrawler

Rule

Path

Disallow

/

linguee

Rule

Path

Disallow

/

linkdexbot

Rule

Path

Disallow

/

linkdexbot/2.2

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linklooker

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkscan/8.1a.unix

Rule

Path

Disallow

/

linkstats

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

lipperhey-kaus-australis

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

lssrocketcrawler

Rule

Path

Disallow

/

ltx71

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

macinroy

Rule

Path

Disallow

/

magpie-crawler

Rule

Path

Disallow

/

mail.ru_bot

Rule

Path

Disallow

/

mail.ru_bot/robots/2.0

Rule

Path

Disallow

/

mass downloader

Rule

Path

Disallow

/

mass downloader/2.2

Rule

Path

Disallow

/

masscan/1.0

Rule

Path

Disallow

/

mata.hari

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

mauibot

Rule

Path

Disallow

/

mauibot (crawler.feedback+wc@gmail.com)

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

mediafox/x.y

Rule

Path

Disallow

/

megaindex.com

Rule

Path

Disallow

/

megaindex.ru

Rule

Path

Disallow

/

megaindex.ru/2.0

Rule

Path

Disallow

/

merzscope

Rule

Path

Disallow

/

metagopher

Rule

Path

Disallow

/

metajobbot

Rule

Path

Disallow

/

microsoft url control

Rule

Path

Disallow

/

microsoft url control - 5.01.4511

Rule

Path

Disallow

/

microsoft url control - 6.00.8169

Rule

Path

Disallow

/

microsoft.url

Rule

Path

Disallow

/

midown tool

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

mj12bot/v1.4.8

Rule

Path

Disallow

/

mindupbot

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mister pix ii 2.01

Rule

Path

Disallow

/

mister pix ii 2.02a

Rule

Path

Disallow

/

mister pix version.dll

Rule

Path

Disallow

/

mister.pix

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

momspider/1.00 libwww-perl/0.40

Rule

Path

Disallow

/

motor/0.2

Rule

Path

Disallow

/

mqqbrowser

Rule

Path

Disallow

/

msiecrawler

Rule

Path

Disallow

/

msr-isrccrawler

Rule

Path

Disallow

/

naver

Rule

Path

Disallow

/

navroad

Rule

Path

Disallow

/

ncbot

Rule

Path

Disallow

/

nearsite

Rule

Path

Disallow

/

neosciocrawler

Rule

Path

Disallow

/

nerdybot

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netants/1.10

Rule

Path

Disallow

/

netants/1.23

Rule

Path

Disallow

/

netants/1.24

Rule

Path

Disallow

/

netants/1.25

Rule

Path

Disallow

/

netcarta cyberpilot pro

Rule

Path

Disallow

/

netcraftsurveyagent/1.0

Rule

Path

Disallow

/

netestate

Rule

Path

Disallow

/

netestate ne crawler

Rule

Path

Disallow

/

netmechanic

Rule

Path

Disallow

/

netscoop/1.0 libwww/5.0a

Rule

Path

Disallow

/

netseer

Rule

Path

Disallow

/

netspider

Rule

Path

Disallow

/

netsystemsresearch

Rule

Path

Disallow

/

net vampire

Rule

Path

Disallow

/

net vampire/3.0

Rule

Path

Disallow

/

netzip

Rule

Path

Disallow

/

netzip downloader 1.0 win32(nov 12 1998)

Rule

Path

Disallow

/

netzip-downloader/1.0.62 (win32; dec 7 1998)

Rule

Path

Disallow

/

netzippy+(http://www.innerprise.net/usp-spider.asp)

Rule

Path

Disallow

/

nhsewalker/3.0

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

nimbostratus-bot/v1.3.2

Rule

Path

Disallow

/

nomad-v2.x

Rule

Path

Disallow

/

npbot

Rule

Path

Disallow

/

nsrbot/1.0

Rule

Path

Disallow

/

nutch

Rule

Path

Disallow

/

obot

Rule

Path

Disallow

/

obot/2.3.1

Rule

Path

Disallow

/

occam/1.0

Rule

Path

Disallow

/

octopus

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

offline explorer/1.2

Rule

Path

Disallow

/

offline explorer/1.4

Rule

Path

Disallow

/

offline explorer/1.6

Rule

Path

Disallow

/

offline explorer/1.7

Rule

Path

Disallow

/

offline explorer/1.9

Rule

Path

Disallow

/

offline explorer/2.0

Rule

Path

Disallow

/

offline explorer/2.1

Rule

Path

Disallow

/

offline explorer/2.3

Rule

Path

Disallow

/

offline explorer/2.4

Rule

Path

Disallow

/

offline explorer/2.5

Rule

Path

Disallow

/

offline navigator

Rule

Path

Disallow

/

offline.explorer

Rule

Path

Disallow

/

ogspider

Rule

Path

Disallow

/

open text site crawler v1.0

Rule

Path

Disallow

/

openbot

Rule

Path

Disallow

/

openhosebot

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openfind data gatherer

Rule

Path

Disallow

/

oracle ultra search

Rule

Path

Disallow

/

pagegrabber

Rule

Path

Disallow

/

papa foto

Rule

Path

Disallow

/

pavuk

Rule

Path

Disallow

/

pcbrowser

Rule

Path

Disallow

/

peoplepal 6.2

Rule

Path

Disallow

/

perman

Rule

Path

Disallow

/

pgp-ka/1.2

Rule

Path

Disallow

/

plista

Rule

Path

Disallow

/

plukkie

Rule

Path

Disallow

/

plukkie/1.6

Rule

Path

Disallow

/

probethenet

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

proximic

Rule

Path

Disallow

/

psbot

Rule

Path

Disallow

/

python/3.5 aiohttp

Rule

Path

Disallow

/

python-requests/2.23.0

Rule

Path

Disallow

/

python-urllib

Rule

Path

Disallow

/

queryn.metasearch

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

qwantify

Rule

Path

Disallow

/

r6_commentreader

Rule

Path

Disallow

/

r6_feedfetcher

Rule

Path

Disallow

/

radiation retriever 1.1

Rule

Path

Disallow

/

realdownload

Rule

Path

Disallow

/

realdownload/4.0.0.40

Rule

Path

Disallow

/

realdownload/4.0.0.41

Rule

Path

Disallow

/

realdownload/4.0.0.42

Rule

Path

Disallow

/

reget

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

/

resume robot

Rule

Path

Disallow

/

riddler

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

rogerbot

Rule

Path

Disallow

/

roverbot

Rule

Path

Disallow

/

sabsimbot

Rule

Path

Disallow

/

safednsbot

Rule

Path

Disallow

/

safetynet robot 0.1

Rule

Path

Disallow

/

searchmetricsbot

Rule

Path

Disallow

/

searchpreview

Rule

Path

Disallow

/

seekport crawler

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

senrigan/xxxxxx

Rule

Path

Disallow

/

sentibot

Rule

Path

Disallow

/

seocompany

Rule

Path

Disallow

/

seodiver

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seoscanners.net

Rule

Path

Disallow

/

serpstatbot/1.0

Rule

Path

Disallow

/

seznambot

Rule

Path

Disallow

/

seznambot/3.2

Rule

Path

Disallow

/

sg-orbiter

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

siteexplorer

Rule

Path

Disallow

/

siteliner

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

slackbot 1.0

Rule

Path

Disallow

/

slackbot-linkexpanding 1.0

Rule

Path

Disallow

/

slack-imgproxy

Rule

Path

Disallow

/

slysearch

Rule

Path

Disallow

/

smartdownload

Rule

Path

Disallow

/

smartdownload/1.2.76 (win32; apr 1 1999)

Rule

Path

Disallow

/

smartdownload/1.2.77 (win32; aug 17 1999)

Rule

Path

Disallow

/

smartdownload/1.2.77 (win32; feb 1 2000)

Rule

Path

Disallow

/

smartdownload/1.2.77 (win32; jun 19 2001)

Rule

Path

Disallow

/

snooper/b97_01

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

sogou web spider/4.0

Rule

Path

Disallow

/

solbot/1.0 lwp/5.07

Rule

Path

Disallow

/

sootle

Rule

Path

Disallow

/

sosospider

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

spanner/1.0 (linux 2.0.27 i586)

Rule

Path

Disallow

/

spiderbot

Rule

Path

Disallow

/

sputnikbot/2.3

Rule

Path

Disallow

/

spyder3.microsys.com

Rule

Path

Disallow

/

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux

Rule

Path

Disallow

/

ssearch

Rule

Path

Disallow

/

startmebot/1.0

Rule

Path

Disallow

/

superbot

Rule

Path

Disallow

/

superbot/3.0 (win32)

Rule

Path

Disallow

/

superbot/3.1 (win32)

Rule

Path

Disallow

/

superhttp

Rule

Path

Disallow

/

superhttp/1.0

Rule

Path

Disallow

/

surdotlybot/1.0

Rule

Path

Disallow

/

surfbot

Rule

Path

Disallow

/

surveybot

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

takeout

Rule

Path

Disallow

/

telegrambot

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleport pro

Rule

Path

Disallow

/

teleport pro/1.29

Rule

Path

Disallow

/

teleport pro/1.29.1590

Rule

Path

Disallow

/

teleport pro/1.29.1634

Rule

Path

Disallow

/

teleport pro/1.29.1718

Rule

Path

Disallow

/

teleport pro/1.29.1820

Rule

Path

Disallow

/

teleport pro/1.29.1847

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

the.intraformant

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

the knowledge ai

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

thumbsniper

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

toweya.com

Rule

Path

Disallow

/

tpradstxtcrawler/1.0

Rule

Path

Disallow

/

trendictionbot

Rule

Path

Disallow

/

trendsmapresolver/0.1

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

true_robot/1.0

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

turnitinbot

Rule

Path

Disallow

/

tweetmemebot/4.0

Rule

Path

Disallow

/

twingly recon-klondike/1.0 (+https://developer.twingly.com)

Rule

Path

Disallow

/

ucbrowser

Rule

Path

Disallow

/

ucsd-crawler

Rule

Path

Disallow

/

um-ic

Rule

Path

Disallow

/

unisterbot

Rule

Path

Disallow

/

unwindfetchor/1.0

Rule

Path

Disallow

/

url control

Rule

Path

Disallow

/

urlspiderpro

Rule

Path

Disallow

/

urlck/1.2.3

Rule

Path

Disallow

/

urly.warning

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

valkyrie/1.0 libwww-perl/0.40

Rule

Path

Disallow

/

vbseo

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

velenpublicwebcrawler

Rule

Path

Disallow

/

voideye

Rule

Path

Disallow

/

waybackarchive.org

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

web.image.collector

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

web sucker

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webauto/3.40 (win98; i)

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webcapture 2.0

Rule

Path

Disallow

/

webcopier

Rule

Path

Disallow

/

webcopier v.2.2

Rule

Path

Disallow

/

webcopier v.2.5

Rule

Path

Disallow

/

webcopier v.2.6

Rule

Path

Disallow

/

webcopier v.2.7a

Rule

Path

Disallow

/

webcopier v.2.8

Rule

Path

Disallow

/

webcopier v.3.0

Rule

Path

Disallow

/

webcopier v.3.0.1

Rule

Path

Disallow

/

webcopier v.3.2

Rule

Path

Disallow

/

webcopier v.3.2a

Rule

Path

Disallow

/

webcopy/

Rule

Path

Disallow

/

webcrawler/3.0 robot libwww/5.0a

Rule

Path

Disallow

/

webemailextrac.*

Rule

Path

Disallow

/

webenhancer

Rule

Path

Disallow

/

webferret

Rule

Path

Disallow

/

webfetch

Rule

Path

Disallow

/

webfetch/2.1.0

Rule

Path

Disallow

/

webfetcher/0.8,

Rule

Path

Disallow

/

webgo is

Rule

Path

Disallow

/

weblayers/0.0

Rule

Path

Disallow

/

webleacher

Rule

Path

Disallow

/

weblinker/0.0 libwww-perl/0.1

Rule

Path

Disallow

/

webmasterworld extractor

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

webmoose/0.0.0000

Rule

Path

Disallow

/

webreaper

Rule

Path

Disallow

/

webreaper [info@webreaper.net]

Rule

Path

Disallow

/

webreaper [webreaper@otway.com]

Rule

Path

Disallow

/

webreaper v9.1 - www.otway.com/webreaper

Rule

Path

Disallow

/

webreaper v9.7 - www.webreaper.net

Rule

Path

Disallow

/

webreaper v9.8 - www.webreaper.net

Rule

Path

Disallow

/

webreaper vwebreaper v7.3 - www,otway.com/webreaper

Rule

Path

Disallow

/

webs@recruit.co.jp

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

websauger 1.20b

Rule

Path

Disallow

/

websauger 1.20j

Rule

Path

Disallow

/

websauger 1.20k

Rule

Path

Disallow

/

website extractor

Rule

Path

Disallow

/

website.quester

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

website quester - www.asona.org

Rule

Path

Disallow

/

website quester - www.esalesbiz.com/extra/

Rule

Path

Disallow

/

webster.pro

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webstripper/2.03

Rule

Path

Disallow

/

webstripper/2.10

Rule

Path

Disallow

/

webstripper/2.12

Rule

Path

Disallow

/

webstripper/2.13

Rule

Path

Disallow

/

webstripper/2.15

Rule

Path

Disallow

/

webstripper/2.16

Rule

Path

Disallow

/

webstripper/2.19

Rule

Path

Disallow

/

webvac

Rule

Path

Disallow

/

webvac/1.0

Rule

Path

Disallow

/

webwalk

Rule

Path

Disallow

/

webwalker

Rule

Path

Disallow

/

webwalker/1.10

Rule

Path

Disallow

/

webwatch

Rule

Path

Disallow

/

webwhacker

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/2.75 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/3.65 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/3.80 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

webzip/4.0 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/4.1 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/4.21

Rule

Path

Disallow

/

webzip/4.21 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/5.0

Rule

Path

Disallow

/

webzip/5.0 (http://www.spidersoft.com)

Rule

Path

Disallow

/

webzip/5.0 pr1 (http://www.spidersoft.com)

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.4.0

Rule

Path

Disallow

/

wget/1.5.2

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

wget/1.7

Rule

Path

Disallow

/

wget/1.8

Rule

Path

Disallow

/

wget/1.8.1

Rule

Path

Disallow

/

wget/1.8.1+cvs

Rule

Path

Disallow

/

wget/1.8.2

Rule

Path

Disallow

/

wget/1.9-beta

Rule

Path

Disallow

/

whowhere robot

Rule

Path

Disallow

/

widow

Rule

Path

Disallow

/

wired-digital-newsbot/1.5

Rule

Path

Disallow

/

woriobot

Rule

Path

Disallow

/

wotbox

Rule

Path

Disallow

/

www collector

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

www.freeloader.com.

Rule

Path

Disallow

/

wwwoffle

Rule

Path

Disallow

/

wwwwanderer v3.0

Rule

Path

Disallow

/

xaldon webspider

Rule

Path

Disallow

/

xaldon webspider 2.5.b3

Rule

Path

Disallow

/

xaldon_webspider

Rule

Path

Disallow

/

xenu's

Rule

Path

Disallow

/

xenu's link sleuth 1.1c

Rule

Path

Disallow

/

xget/0.7

Rule

Path

Disallow

/

xovibot/2.0

Rule

Path

Disallow

/

yacybot

Rule

Path

Disallow

/

yandexdirect

Rule

Path

Disallow

/

yasaklibot

Rule

Path

Disallow

/

yes

Rule

Path

Disallow

/

yesupbot

Rule

Path

Disallow

/

yeti

Rule

Path

Disallow

/

yisouspider

Rule

Path

Disallow

/

youdaobot

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zeus 11389 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 11652 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 18018 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 26378 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 30747 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 39206 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 41641 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 44238 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 51070 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 51674 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 51837 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 63567 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 6694 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 82016 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 82900 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 84842 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 90872 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 94934 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 95245 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 95351 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus 97371 webster pro v2.9 win32

Rule

Path

Disallow

/

zeus link scout

Rule

Path

Disallow

/

zyborg

Rule

Path

Disallow

/

Site Domain	rust.co.uk
Base Domain	rust.co.uk
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Couldn't establish SSL connection.
Last Scan	2025-12-11T05:55:50+00:00
Next Scan	2026-02-09T05:55:50+00:00

Scanned	2025-09-19T06:19:43+00:00
URL	https://rust.co.uk/robots.txt
Domain IPs	52.16.139.166
Response IP	52.16.139.166
Found	Yes
Hash	d81a46772081931614eb5f0d3d3ecf3363032ab13db2e7fb281ab1c9e75a83bc
SimHash	72e142a3c0ee

rust.co.ukrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

a6-indexer

abonti

adidxbot

adscanner

ahrefsbot

ahrefsbot

ahrefsbot/6.1

aihitbot

aitcsrobot/1.1

alexa

alexa toolbar

alexibot

alphabot/3.2

alphaseobot

amznkassocbot

apache-httpclient/4.5.2 (java/1.8.0_161)

apache-httpclient/4.5.9 (java/1.8.0_222)

aqua_products

arachnophilia

archive.org_bot

aspider/0.09

aspiegelbot

asterias

auresys/1.0

awariosmartbot/1.0

b2w/0.1

baiduspider

backdoorbot

backdoorbot/1.0

backlinkcrawler

backrub/.

big brother

bizbot003

bizbot04 kirk.overleaf.com

blackboard safeassign

black.hole

black hole

blackwidow

blexbot

blexbot/1.0

blowfish

blowfish/1.0

bookmark search tool

bot mailto:craftbot@yahoo.com

botalot

botrighthere

bspider/1.0 libwww-perl/0.40

bubing

builtbottough

bullseye

bullseye/1.0

bunnyslippers

cactvs chemistry spider

careerbot

ccbot

cegbfeieh

changedetection

checkbot/x.xx lwp/5.x

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

chinaclaw

cliqzbot

cloudservermarketspider

coccoc

coccocbot-image/1.0

coibotparser/2.0

combine/0.0

compspybot

conceptbot/0.3

copernic

copyrightcheck

cosmos

rust.co.uk
robots.txt