asgnet.co.uk
robots.txt

Groups

ahc

www.aihitdata.com

anthropic

ahrefsbot

amazonbot

applebot

archive.org_bot

bananabot

barkrowler

blexbot

borneobot

censysinspect

claudebot

clickagy

cliqzbot

crawler.007ac9.net

crawler4j

checkmarknetwork

chrome-lighthouse

dataprovider

domaincrawler

domainsproject.org

downloaderchrome

evc-batch

exabot

facebook

facebookexternalhit

facebot

fedora

www.grapeshot.co.uk

gptbot

grover

hubspot

idg/uk

jersey

komodiabot

liebaofast

lightspeedsystemscrawler

ltx71

masscan

mauibot

maxpointcrawler

megaindex

megaindex.ru

mj12bot

neevabot

nutch

pagething

panscient

pleskbot

pocketparser_ua

safednsbot

scrapy

seekport

semrushbot

seokicks-robot

serpstatbot

sogou

spaziodati

special_archiver

surdotlybot

twitterbot

uptimebot

webprosbot

wellknownbot

wotbox

baiduspider

cipacrawler

dotbot

ioncrawl

jobboersebot

linkdex.com

mtrobot

obot

pagething

qwantify

rogerbot

searchmetricsbot

sirdatabot

spbot

surveybot

netestate ne crawler

verity

voltron

wget/1.9

yandex

xovionpagecrawler

yandexbot

zoominfobot

orgprobe

orgprobe/2.0.0

netcraftsurveyagent

http://www.nominet.org.uk/privacypolicy

plukkie

trendictionbot

siteexplorer

dispatch

garlikcrawler

extlinksbot

ccbot

criteobot/0.1

duckduckbot

Other Records

thetradedesk

googlebot

Comments

• seems to obey robots.txt
• does not honor
• does not honour robots.txt
• - hacking bot (OVH ?)
• github - not wanted - Seems to obey
• all sources found (not many) say bad
• chrome lighthouse bad extension/bot - ignores disallow
• ignores disallow
• obeys robots.txt - opensiteexplorer.org
• WAS a good bot - now exabot.com unreachable
• on .jp server since 28-12-2018 and generating 400 errors so disallow
• no facebook crawler - don't want it, need it or anything to do with it - Also ignores robots.txt
• obeys robots.txt
• MICROSOFT OPEN AI - PERM BAN ALL AI SCUM
• hubspot - excess scrapping - now ignores robots.txt
• jersey does not appear to obey robots.txt (amazonAWS)
• bad bot blocked at system level as well
• reported bad bot(s) on github
• bad bot ? - if ignores this one - block at app level - 05/03/2018
• BAD - does not even read robots.txt - Banned at application level
• obeys robots.txt
• Nutch ignores robots.txt - blocked
• PITA - Banned at higher Level
• unconfirmed Plesk bot - ban until confirmed or denied by Plesk
• bad bot
• Bad bot - No Robots.txt read
• bad bot - only reads robots AFTER being blocked and ignores anyway
• obeys robots.txt - Sometimes
• bad bot block at firewall level
• Bad bot ignores robots.txt
• spaziodati.eu (see also IDG/UK bot) - ignores both disallows
• bad bot
• Web spam - trying to con real uptimerobots users - Ignores this setting
• bots that obey or read robots.txt but not wanted or ignore settings below
• openlinkprofiler.org
• domainTools
• orgprobe - suspect - reported obeys and also ignores robots.txt - lets find out shall we
• netcraft survey agent - doesn't read robots.txt as far as we can tell
• if passes this then block at firewall level as bad bot
• also netcraft web server survey but no bot identifiable - ban all UA with netcraft
• at app firewall
• nominet no longer simply reads site status but tries hacking WP, Joomla etc so
• block as per their rules (March 2018) - awaiting policy changes (April 2018)
• plukkie (ykoon)
• .de social networking bot - not wanted or needed
• siteexplorer (back link checkeer) - disallow
• seems to obey robots.txt
• SSL problem on parent site (08-11-2018) now blocked
• ALLOWS and Partials Start Here for Good Bots
• proximic - reallow to PAS - Aug 2022
• User-agent: proximic
• Disallow:
• ccbot - commercial Listed as Good and honours robot.txt from AWS - Disallow members pas

Warnings

• 2 invalid lines.