forum.starmen.net
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

*

*

*

mj12bot

ccbot

chatgpt-user

gptbot

google-extended

anthropic-ai

omgilibot

omgili

facebookbot

imagesiftbot

meta-externalagent

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• GENERAL NOTICE TO ALL BOTS
• If for some strange reason your bot wants to login and do things, that's cool.
• However, your bot MUST respect session cookies like a normal browser. Else, I will prevent your bot from getting a cookie.
• Due to these user agents below not following standard cookies, the following user agents are prevented from getting a session, therefore a cookie, and therefore able to login.
• This is not a punishment so much as just prevent clogging the session store of you search engines that don't care about cookies, so why waste the space? :)
• 'msnbot', # Microsoft Bing search engine
• 'yahoo! slurp', # Yahoo! search engine
• 'googlebot', # Google search engine
• 'mediapartners', # Google AdSense spider
• 'feedfetcher-google', # Google Feedfetcher RSS fetcher
• 'teoma', # Ask Jeeves search engine
• 'wordpress', # Wordpress pingbacks
• 'baiduspider', # Chinese search/MP3 engine
• 'sparkflare', # Sparkflare feed fetcher for Campfire
• 'bingbot', # Bing changed
• 'downscout',
• 'ltx71',
• 'magpie-crawler', #Brandwatch? Dunno but they are MASSIVELY annoying and 100 times bigger than the next bot
• 'ahrefsbot',
• 'oauth',
• '360spider',
• 'psbot'
• See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
• To ban all spiders from the entire site uncomment the next two lines:
• User-Agent: *
• Disallow: /

Warnings

• `content-signal` is not a known field.