semacraft.com
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

*

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• If the Joomla site is installed within a folder such as at
• e.g. www.example.com/joomla/ the robots.txt file MUST be
• moved to the site root at e.g. www.example.com/robots.txt
• AND the joomla folder name MUST be prefixed to the disallowed
• path, e.g. the Disallow rule for the /administrator/ folder
• MUST be changed to read Disallow: /joomla/administrator/
• For more information about the robots.txt standard, see:
• http://www.robotstxt.org/orig.html
• For syntax checking, see:
• http://www.sxw.org.uk/computing/robots/check.html

Warnings

• `content-signal` is not a known field.