buxtons.net
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

*

-ai
_ai
ai.
ai-
ai_
ai=
addsearchbot
agentic
agentql
agent 3
agent api
ai agent
ai article writer
ai chat
ai content detector
ai detection
ai dungeon
ai journalist
ai legion
ai rag
ai search
ai seo crawler
ai training
ai web
ai writer
ai2
aibot
aihitbot
aimatrix
aisearch
aitraining
alexa
alice yandex
aligenie
aliyunsec
alpha ai
alphaai
amazon
amelia
anderspinkbot
andibot
anonymous ai
anthropic
anypicker
anyword
applebot
aria ai
aria browse
articoolo
ask ai
autogen
autoglm
automated writer
automl
autonomous rag
awariorssbot
awariosmartbot
aws trainium
azure
babyagi
babycatagi
bardbot
basic rag
bedrock
big sur
bigsur
botsonic
brightbot
browser mcp agent
browser use
bytebot
bytedance
bytespider
carynai
catboost
cc-crawler
ccbot
chai
character
charstar ai
chatbot
chatglm
chatsonic
chatuser
chinchilla
claude
clearscope
clearview
cognitive ai
cohere
common crawl
commoncrawl
content harmony
content king
content optimizer
content samurai
contentatscale
contentbot
contentedge
contentshake
conversion ai
copilot
copyai
copymatic
copyscape
coreweave
corrective rag
cotoyogi
crab
crawl4ai
crawlq ai
crawlspace
crew ai
crewai
crushon ai
dall-e
darkbard
datafor
dataprovider
datenbank crawler
deepai
deep ai
deepl
deepmind
deep research
deepresearch
deepseek
devin
diffbot
doubao ai
duckassistbot
duckduckgo chat
duckduckgo-enhanced
echobot
echobox
elixir
facebookbot
facebookexternalhit
factset
falcon
fire-1
firebase
firecrawl
flux
flyriver
frase ai
friendlycrawler
gato
gemini
gemma
gen ai
genai
generative
genspark
gentoo-chat
ghostwriter
gigachat
glm
godmode
goose
gpt
grammarly
grendizer
grok
gt bot
gtbot
gtp
hemingway editor
hetzner
hugging
hunyuan
hybrid search rag
hypotenuse ai
iask
icc-crawler
imagegen
imagesiftbot
img2dataset
imgproxy
ink editor
inkforall
instructor
intelliseek
inferkit
isscyberriskcrawler
janitor ai
jasper
jenni ai
julius ai
kafkai
kaggle
kangaroo
keyword density ai
kimi
knowledge
komobot
kruti
langchain
le chat
lensa
lightpanda
linerbot
llama
llm
local rag agent
lovable
magistral
magpie-crawler
manus
marketmuse
meltwater
meta-ai
meta-external
meta-webindexer
meta ai
metaai
metatagbot
middleware
midjourney
mini agi
minimax
mintlify
mistral
mixtral
model-training
monica
narrative
neevabot
netestate
neural text
neuralseo
ninjaai
nodezero
nova act
novaact
oai-searchbot
oai searchbot
oasis
olivia
omgili
open ai
open interpreter
openagi
openai
openbot
openpi
openrouter
opentext ai
operator
outwrite
page analyzer ai
pangubot
panscient
paperlibot
paraphraser.io
peer39_crawler
perflexity
perplexity
petal
phind
piplbot
poebot
poesearchbot
prowritingaid
proximic
puppeteer
python ai
qualified
quark
quillbot
qopywriter
qwen
rag agent
rag azure ai
rag chatbot
rag database
rag is
rag pipeline
rag search
rag with
rag-
rag_
raptor
react agent
redis ai rag
robotspider
rytr
saplingai
sbintuitionsbot
scala
scalenut
scrap
scriptbook
seekr
seobot
seo content machine
seo robot
semrushbot
sentibot
serper
shapbot
sidetrade
simplified ai
sitefinity
skydancer
slickwrite
smartbot
sonic
sora
spider/2
spidercreator
spin rewrite
spinbot
stability
stablediffusionbot
sudowrite
summalybot
super agent
superagent
superagi
surfer ai
terracotta
text blaze
textcortex
thinkbot
thordata
tiktokspider
timpibot
tinybird
together ai
traefik
turnitinbot
uagents
velenpublicwebcrawler
venus chub ai
vidnami ai
vision rag
websurfer
webtext
webzio
wechat
whisper
wordai
wordtune
wpbot
writecream
writerzen
writescope
writesonic
xai
xbot
yaml
yandexadditional
youbot
zendesk
zero
zhipu
zhuque ai
zimm

gptbot

chatgpt-user

ccbot

anthropic-ai

claude-web

cohere-ai

omgilibot

omgili

facebookbot

meta-externalagent

diffbot

bytespider

imagesiftbot

yandexbot

googlebot

bingbot

slurp

duckduckbot

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• Allow all other bots full access
• Block AI bots from all access
• Block OpenAI GPTBot
• Block ChatGPT User Agent
• Block Common Crawl Bot (used to train AI models)
• Block Anthropic AI (Claude)
• Block Cohere AI
• Block Omgili Crawler
• Block Facebook Bot (used for AI training)
• Block Diffbot
• Block ByteDance (TikTok) Spider
• Block ImagesiftBot
• Block Yandex Bot (optional - may affect Russian search visibility)
• ========================================
• Allow legitimate search engine bots
• ========================================
• Make sure these are still allowed:

Warnings

• 1 invalid line.
• `content-signal` is not a known field.