manjaro.news
robots.txt

Groups

admantx
ai2bot
ai2bot-dolma
aibot
alexa
alexibot
alittleclient
alligator
allsubmitter
alphabot
amazonbot
anarchie
anarchy
anarchy99
ankit
anthill
anthropic-ai
anthropic-ai
apache-httpclient
apexoo
applebot-extended
aspiegel
asterias
atomseobot
attach
awariobot
awariorssbot
awariosmartbot
backdoorbot
backlink-ceck
backlink-check
backlinkcrawler
backlinksextendedbot
backstreet
backweb
badass
bandit
barkrowler
batchftp
battleztarbazinga
bbbike
bdcbot
bdfetch
betabot
bigfoot
bitacle
blackboard
blackhole
blackwidow
blazer
blexbot
blow
blowfish
boardreader
bolt
botalot
brandprotect
brandwatch
brightbot 1.0
bubing
buck
buddy
builtbottough
builtwith
bullseye
bunnyslippers
buzzsumo
bytespider
cah.io.community
calculon
catexplorador
cazoodlebot
ccbot
cegbfeieh
censysinspect
chatgpt-user
cheesebot
cherrypicker
cheteam
chinaclaw
chlooe
chrome privacy preserving prefetch proxy
citoid
claritybot
clark-crawler
claudebot
claude-web
cliqzbot
cloudmapping
coccocbot
cocolyzebot
code87
cognitiveseo
coher-ai
cohere-ai
cohere-training-data-crawler
collector
com.plumanalytics
copier
copyrightcheck
copyscape
cosmos
craftbot
crawler4j
crawler.feedback
crawlingathomeproject
crawl.sogou.com
crawlspace
crazywebcrawler
crescent
crunchbot
cshttp
curious
custo
cyberfind
cyotekwebcopy
dart
databasedrivermysqli
datacha0s
dataforseobot
dataforseo.com
dblbot
demandbase-bot
demon
deusu
devil
diffbot
digincore
digitalpebble
diibot
dirbuster
disco
discobot
discoverybot
dispatch
dittospyder
dnbcrawler-analytics
dnyzbot
domainappender
domaincrawler
domainsigmacrawler
domainsproject
domainstatsbot
domcopbot
dotbot
downloadwonder
dragonfly
drip
duckassistbot
easydl
ebingbong
ecatch
eccp/1.0
ecxi
eirgrabber
emailsiphon
emailwolf
erocrawler
evc-batch
ev-crawler
evil
exabot
expanse
expresswebpictures
extlinksbot
extractor
extractorpro
extremepicturefinder
eyenetie
ezooms
facebookbot
facebookexternalhit
facebookplatform
facebookscraper
fdm
femtosearchbot
fhscan
fimap
firefox/7.0
flashget
flunky
foobot
freeuploader
friendlycrawler
frontpage
fuzz
fyberspider
fyrebot
galaxybot
geedoproductsearch
genieo
germcrawler
getintent
getright
getweb
g-i-g-a-b-o-t
gigabot
go-ahead-got-it
go-http-client
google-extended
googleother
googleother-image
googleother-video
gopher
gotit
go!zilla
gozilla
gptbot
grabber
grabnet
grafula
grapefx
grapeshotcrawler
grfzbot
gridbot
gt::www
haansoft
haosouspider
harvest
havij
heritrix
hloader
honolulubot
http-client
httpoison
http.rb
humanlinks
hybridbot
iaskspider/2.0
iblog
icc-crawler
idbot
idbte4m
id-search
ilsebot
imagefetch
imagesiftbot
imagesift.com
img2dataset
imgproxy
indeedbot
indylibrary
infonavirobot
informationsecurityteaminfrasecscanner
infotekies
infrasecscanner
instabid
intelliseek
interget
internetmeasurement
internetninja
internetseer
internetvista monitor
iria
irlbot
isitwp.com
iskanie
isscyberriskcrawler
istellabot
iubenda-radar
jamesbot
java
jbrofuzz
jennybot
jetcar
jetty
jikespider
jocwebspider
joomla
jorgee
justview
jyxobot
kangaroo bot
kenjinspider
keybottranslation-search-machine
keycdn-tools
keyworddensity
kinza
kozmosbot
lanshanbot
larbin
leap
leechftp
leechget
lexibot
lftp
libweb
libwhisker
libwww-perl
liebaofast
lightspeedsystems
likse
linkbot
linkdexbot
linkedinbot
linkextractorpro
linkfluence
linkpadbot
linkscan
linksmanager
linkwalker
linqiametadatadownloaderbot
linqiarssbot
linqiascrapebot
lipperhey
lipperheyspider
litemage_walker
lmspider
lnspiderguy
ltx71
lwp::simple
magnet
mag-net
magpie-crawler
mail.ru_bot
majestic12
majestic-seo
majesticseo
markmonitor
markwatch
masscan
massdownloader
matahari
mauibot
mb2345browser
meanpathbot
meanpathbot
mediatoolkitbot
mediawords
megaindex.ru
meta-externalagent
meta-externalfetcher
metauri
micromessenger
microsoftdataaccess
microsofturlcontrol
minefield
misterpix
mobliesafari
mojolicious
molokaibot
moptimizer
morfeusfuckingscanner
mozlila
mr.4x3
msrabot
muhstik-scan
musobot
nameintelligence
nameprotect
navroad
nearsite
needle
nessus
netants
netcraft
netestate ne crawler
netlyzer
netmechanic
netspider
nettrack
netvampire
netvibes
netzip
nextgensearchbot
nibbler
nicerspro
niki-bot
nikto
nimblecrawler
nimbostratus
ninja
nmap
node-fetch
novaact
npbot
nuclei
nutch
oai-searchbot
obot
odin
offlineexplorer
offlinenavigator
omgili
omgili
omgilibot
oncrawl
openai
openai.com
openfind
openlinkprofiler
openvas
operator
orangebot
orangespider
outclicksbot
outfoxbot
page analyzer
pagegrabber
pagescorer
pagething.com
palmsource
pandalytics
pangubot
panscient
papafoto
pavuk
pcbrowser
peoplepal
perplexitybot
perplexity-user
petalbot
petalbot
php
picscout
picsearch
picturefinder
piepmatz
pimonster
pinterestbot
pixray
pleasecrawl
pockey
polaris version
probe-image-size
probethenet
propowerbot
prowebwalker
proximic
psbot
pu_in
pump
pxbroker
pycurl
python-requests
python-urllib
querynmetasearch
quick-crawler
quora-bot
r6_commentreader
r6_feedfetcher
rainbot
rankactive
rankactivelinkbot
rankflex
rankingbot
rankingbot2
rankivabot
rankurbot
realdownload
reaper
rebelmouse
recorder
red
redesscrapy
reget
repomonkey
reqwest
rere
researchscan
ripper
ripz
rocketcrawler
rogerbot
rpt-httpclient
rssingbot
ruby
s1z.ru
sbider
scalaj-http
scan.lol
scrapy
semrushbot-ocob
semrushbot-swa
sentibot
senutobot
seobility
seobilitybot
seocherrybot
seocompany.store
seomoz
seoscanners
seositecheckup
seostar
serpstatbot
sexsearcher
sffeedreader
shodan
sidetrade indexer bot
siphon
slackbot
slackbot-linkexpanding
slack-imgproxy
slysearch
smartdownload
snake
snapbot
snoopy
socialrankiobot
sociscraper
sogouspider
sogouwebspider
sosospider
sottopop
spacebison
spammen
spankbot
spanner
sp_auditbot
spbot
spider_bot
spider_bot/3.0
spinn3r
splitsignalbot
sputnikbot
sqlmap
sqlworm
sqworm
steeler
stripper
sucker
sucuri
summalybot
superbot
superhttp
surdotlybot
surfbot
surveybot
suzuran
swiftbot
szukacz
t0phackteam
t8abot
takeout
telegrambot
teleport
teleportpro
telesoft
telesphoreo
telesphorep
theintraformant
thenomad
thumbor
tighttwatbot
tiktok
timpibot
tineye
tinytestbot
titan
toata
toweyabot
tracemyfile
trendiction
trendictionbot
trendiction.com
trendiction.de
true_robot
turingos
turnitin
turnitinbot
twengabot
twice
twitterbot
typhoeus
ubermetrics-technologies.com
unisterbot
upflow
urly.warning
urlywarning
vacuum
vagabondo
v-bot
vbproject
vci
velenpublicwebcrawler
vericitecrawler
vidiblescraper
virusdie
voideye
voil
voltron
voyagerx.com
wallpapers
wallpapers/3.0
wallpapershd
wasalive-bot
wbsearchbot
webalta
webauto
webbandit
webcollage
webcopier
webdav
webenhancer
webfetch
webfuck
webgains-bot
webgois
webimagecollector
webleacher
webmasterworldforumbot
webmeup-crawler
webpix
webprosbot
webpros.com
webreaper
websauger
webshag
websiteextractor
websitequester
webster
webstripper
websucker
webvigil
webwhacker
webzio-extended
webzip
wesee
whack
whacker
whatsapp
whatweb
who.isbot
widow
winhttp.winhttprequest
winhttrack
wiseguysrobot
wisenutbot
wonderbot
woobot
wotbox
wprecon
wpscan
www-collector-e
www-mechanize
www::mechanize
wwwoffle
x09mozilla
x22mozilla
xaldon_webspider
xaldonwebspider
xenu
xpymep1.exe
yadirectfetcher
yak
yandexaccessibilitybot
yandexcalendar
yandexcombot
yandexdialogs
yandexdirect
yandexdirectdyn
yandexfavicons
yandexmarket
yandexmetrika
yandexmobilebot
yandexmobilescreenshotbot
yandexontodbapi
yandexpartner
yandexrca
yandexrenderresourcesbot
yandexscreenshotbot
yandexsearchshop
yandextracker
yandexuserproxy
yandexvideoparser
youbot
youdaobot
zade
zauba
zauba.io
zermelo
zeus
zgrab
zitebot
zmeu
zoombot
zoominfobot
zumbot
zyborg

*

Warnings

• 5 invalid lines.