nosubject.com
robots.txt

Groups

adsbot
adscanner
ahrefsbot
aihitbot
anthropic-ai
backlinkcrawler
barkrowler
barkrowler
blackwidow
blexbot
bubing
buck
bytespider
careerbot
cazoodlebot
ccbot
chatgpt-user
checkmarknetwork
chinaclaw
cincraw
cliqzbot
cloudservermarketspider
coccoc
cohere-ai
crawler4j
custo
dataforseobot
df
df bot
disco
domain re-animator bot
domaincrawler
domainstatsbot
dotbot
dotbot
dotbot
dotbot/1.0
download demon
ecatch
eirgrabber
emailsiphon
emailwolf
exabot
express webpictures
extractorpro
eyenetie
facebookbot
flashget
fr-crawler
garlikcrawler
getright
getweb!
gigabot
go-ahead-got-it
go-http-client
go!zilla
google-extended
gosign-security-crawler
gptbot
grabnet
grafula
haosouspider
hmview
httrack
i-market-bot
iccrawler
image stripper
image sucker
implisensebot
indeedbot
indy library
inetdex-bot
interget
internet ninja
jetcar
jobboersebot
jobs.de-robot
joc web spider
kraken
larbin
leechftp
linguee
linkstats
lipperhey-kaus-australis
ltx71
magpie-crawler
mass downloader
meanpathbot
megaindex.com
megaindex.ru
metajobbot
midown tool
mindupbot
mister pix
mj12bot
navroad
nearsite
net vampire
netants
netestate
netspider
netzip
node/simplecrawler
nuclei
obot
octopus
offline explorer
offline navigator
omgili
omgilibot
openhosebot
pagegrabber
papa foto
pavuk
pcbrowser
petalbot
plista
plukkie
python/3.5 aiohttp
qwantify
r6_commentreader
realdownload
reget
riddler
rogerbot
rogerbot
sabsimbot
safednsbot
screaming frog seo spider
searchmetricsbot
seekportbot
semrushbot
semrushbot-sa
sentibot
sentibot
seodiver
seokicks-robot
seoscanners.net
serendeputybot
serpstatbot
sg-orbiter
sidetrade
sistrix
sistrix
sistrix crawler
siteliner
sitesnagger
smartdownload
spbot
spiderbot
startmebot
superbot
superhttp
surfbot
surveybot
t3versionsbot
takeout
teleport pro
thumbsniper
toweya.com
trendictionbot
trendictionbot
turnitinbot
um-ic
unisterbot
velenpublicwebcrawler
voideye
wbsearchbot
web image collector
web sucker
webauto
webcopier
webfetch
webgo is
webleacher
webreaper
websauger
website extractor
website quester
webstripper
webwhacker
webzip
wget
widow
wikido
wotbox
ws-bot-v1
wwwoffle
xaldon webspider
xing
xing bot
yak
yandexbot
zeus
zoominfobot
zoominfobot

Other Records

Comments

• ——————————–
• Ban bots that don’t benefit us.

Warnings

• 3 invalid lines.