femitheist.net
robots.txt

Groups

*

amazonbot

applebot-extended

bytespider

ccbot

claudebot

google-extended

gptbot

meta-externalagent

*

googlebot-image

yandexbot

ahrefsbot

seokicks-robot

sistrix crawler

uptimerobot/2.0

ezooms robot

perl lwp

blexbot

netestate ne crawler (+http://www.website-datenbank.de/)

wiseguys robot

turnitin robot

turnitinbot

turnitin bot

turnitinbot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

turnitinbot/3.0

heritrix

pimonster

pimonster

eccp/1.0 (search@eniro.com)

baiduspider
baiduspider-video
baiduspider-image
mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/3.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/4.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/5.0; +http://www.baidu.com/search/spider.html)
baiduspider/2.0
baiduspider/3.0
baiduspider/4.0
baiduspider/5.0

sogou spider

youdaobot

gsa-crawler (enterprise; t4-knhh62cdkc2w3; gsa_manage@nikon-sys.co.jp)

megaindex.ru/2.0

megaindex.ru

megaindex.ru

mail.ru_bot/2.0

mail.ru

mail.ru_bot/2.0; +http://go.mail.ru/help/robots

mj12bot

mj12bot/v1.4.3

ubicrawler

doc

zao

twiceler

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

nutch

spock

omniexplorer_bot

becomebot

geniebot

dotbot

mlbot

linguee bot

aihitbot

exabot

sbider/nutch

jyxobot

magent

speedy spider

shopwiki

huasai

datacha0s

baiduspider

atomic_email_hunter

mp3bot

winhttp

betabot

core-project

panscient.com

java

libwww-perl

Comments

• As a condition of accessing this website, you agree to abide by the following
• content signals:
• (a) If a content-signal = yes, you may collect content for the corresponding
• use.
• (b) If a content-signal = no, you may not collect content for the
• corresponding use.
• (c) If the website operator does not include a content signal for a
• corresponding use, the website operator neither grants nor restricts
• permission via content signal with respect to the corresponding use.
• The content signals and their meanings are:
• search: building a search index and providing search results (e.g., returning
• hyperlinks and short excerpts from your website's contents). Search does not
• include providing AI-generated search summaries.
• ai-input: inputting content into one or more AI models (e.g., retrieval
• augmented generation, grounding, or other real-time taking of content for
• generative AI search answers).
• ai-train: training or fine-tuning AI models.
• ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
• RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
• AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
• BEGIN Cloudflare Managed content
• END Cloudflare Managed Content
• Google Image Crawler Setup - having crawler-specific sections makes it ignore generic e.g *
• User-agent: Googlebot
• Yandex tends to be rather aggressive, may be worth keeping them at arms lenght
• User-agent: Pinterest
• Crawlers Setup
• User-agent: *
• Block Ahrefs
• Block SEOkicks
• Block SISTRIX
• Block Uptime robot
• Block Ezooms Robot
• Block Perl LWP
• Block BlexBot
• Block netEstate NE Crawler (+http://www.website-datenbank.de/)
• Block WiseGuys Robot
• Block Turnitin Robot
• Block Heritrix
• Block pricepi
• Block Searchmetrics Bot
• User-agent: SearchmetricsBot
• Disallow: /
• Block Eniro
• Block Baidu
• Block SoGou
• Block Youdao
• Block Nikon JP Crawler
• Block MegaIndex.ru
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites or download them for offline viewing. Please obey robots.txt.

Warnings

• 4 invalid lines.
• `content-signal` is not a known field.