blog.stoerdebegga.org
robots.txt

Groups

*

ai2bot
addsearchbot
ai2bot-dolma
amazonbot
andibot
applebot
applebot-extended
awario
brightbot 1.0
bytespider
ccbot
chatgpt agent
chatgpt-user
claude-searchbot
claude-user
claude-web
claudebot
cloudvertexbot
cotoyogi
crawlspace
datenbank crawler
deepseekbot
devin
diffbot
duckassistbot
echobot bot
echoboxbot
facebookbot
factset_spyderbot
firecrawlagent
friendlycrawler
gptbot
gemini-deep-research
google-cloudvertexbot
google-extended
google-firebase
googleagent-mariner
googleother
googleother-image
googleother-video
icc-crawler
isscyberriskcrawler
imagesiftbot
kangaroo bot
linerbot
meta-externalagent
meta-externalfetcher
mistralai-user
mistralai-user/1.0
mycentralaiscraperbot
novaact
oai-searchbot
openai
operator
pangubot
panscient
perplexity-user
perplexitybot
petalbot
phindbot
poseidon research crawler
qualifiedbot
quillbot
sbintuitionsbot
scrapy
semrushbot-ocob
semrushbot-swa
shapbot
sidetrade indexer bot
terracotta
thinkbot
tiktokspider
timpibot
velenpublicwebcrawler
wardbot
webzio-extended
yak
yandexadditional
yandexadditionalbot
youbot
aihitbot
anthropic-ai
bedrockbot
bigsur.ai
cohere-ai
cohere-training-data-crawler
facebookexternalhit
iaskspider/2.0
img2dataset
meta-externalagent
meta-externalfetcher
meta-webindexer
netestate imprint crawler
omgili
omgilibot
panscient.com
quillbot.com
wpbot

Other Records

Comments

• hello robots [^_^]
• let's be friends
• except for these fuckers which are not friends
• (bad bots bundle by https://github.com/ai-robots-txt/robots.json)