citizenwiki.cn
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

mj12bot

mediapartners-google*

israbot

orthogaffe

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

fast

wget

grub-client

k2spider

npbot

webreaper

ahrefsbot

dotbot

mauibot

petalbot

semrushbot

dataforseobot

neevabot

megaindex

seekport

barkrowler

*

Other Records

Other Records

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• _____ _ _____ _ _ _ _____ _
• / ___| | / __ (_) | (_) |_ _| | |
• \ `--.| |_ __ _ _ __ | / \/_| |_ _ _______ _ __ | | ___ ___ | |___
• `--. \ __/ _` | '__| | | | | __| |_ / _ \ '_ \ | |/ _ \ / _ \| / __|
• /\__/ / || (_| | | | \__/\ | |_| |/ / __/ | | | | | (_) | (_) | \__ \
• \____/ \__\__,_|_| \____/_|\__|_/___\___|_| |_| \_/\___/ \___/|_|___/
• Are you a human? Maybe you want to help with the wiki? Contact us on Discord: https://discord.com/invite/XcKwqyD4sc
• robots.txt for Star Citizen Wiki
• based on version from Wikipedia in 2018-04-22
• Please note: There are a lot of pages on this site, and there are
• some misbehaved spiders out there that go _way_ too fast. If you're
• irresponsible, your access to the site may be blocked.
• Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
• and ignoring 429 ratelimit responses, claims to respect robots:
• http://mj12bot.com/
• advertising-related bots:
• Wikipedia work bots:
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Misbehaving: requests much too fast:
• Sorry, wget in its recursive mode is a frequent problem.
• Please read the man page and use it properly; there is a
• --wait option you can use to set the delay between hits,
• for instance.
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• Hits many times per second, not acceptable
• http://www.nameprotect.com/botinfo.html
• A capture bot, downloads gazillions of pages with no public benefit
• http://www.webreaper.net/
• Crawler list from runescape.wiki
• SEO bot
• SEO bot
• Unknown
• Huawei's web crawler. Ignores Disallow and gets caught in loops accessing special pages. Produces the majority of uncachable requests.
• SEO bot
• SEO bot
• SEO bot
• SEO bot
• SEO bot
• SEO bot
• Added rules from SCW

Warnings

• `content-signal` is not a known field.