ttadev.org
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

anthropic-ai

claudebot

claude-web

chatgpt-user

gptbot

perplexitybot

cohere-ai

google-extended

facebookbot

meta-externalagent

timpibot

friendlycrawler

image2dataset

imagesiftbot

ahrefsbot

barkrowler

blexbot

mj12bot

dataforseobot

dotbot

semrushbot

yandex

petalbot

amazonbot

bytespider

censysinspect

expanse

internet-measurement

scrapy

python-requests

java

go-http-client

news-please

dataprovider

orbbot

ioncrawl

isscyberriskcrawler

velenpublicwebcrawler

peer39_crawler

zoominfobot

wp_is_mobile

fast

wget

grub-client

*

Other Records

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• robots.txt for http://www.tunearch.org/ and friends
• Please note: There are a lot of pages on this site, and there are
• some misbehaved spiders out there that go _way_ too fast. If you're
• irresponsible, your access to the site may be blocked.
• Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
• and ignoring 429 ratelimit responses, claims to respect robots:
• http://mj12bot.com/
• === AI CRAWLERS ===
• === SEO CRAWLERS ===
• === SEARCH ENGINES & DATA ===
• === SECURITY & NETWORK SCANNERS ===
• === GENERIC / ABUSIVE ===
• Misbehaving: requests much too fast:
• Sorry, wget in its recursive mode is a frequent problem.
• Please read the man page and use it properly; there is a
• --wait option you can use to set the delay between hits,
• for instance.
• The 'grub' distributed client has been *very* poorly behaved.
• Friendly, low-speed bots are welcome viewing article pages, but not
• dynamically-generated pages please.
• Inktomi's "Slurp" can read a minimum delay between hits; if your
• bot supports such a thing using the 'Crawl-delay' or another
• instruction, please let us know.
• There is a special exception for API mobileview to allow dynamic
• mobile web & app views to load section content.
• These views aren't HTTP-cached but use parser cache aggressively
• and don't expose special: pages etc.
• Another exception is for REST API documentation, located at
• /api/rest_v1/?doc.

Warnings

• `content-signal` is not a known field.