lup.be
robots.txt

Groups

*

No rules defined. All paths allowed.

Other Records

*

ai2bot
ai2bot-dolma
aihitbot
amazonbot
andibot
anthropic-ai
applebot
applebot-extended
brightbot 1.0
bytespider
ccbot
chatgpt-user
claude-searchbot
claude-user
claude-web
claudebot
cohere-ai
cohere-training-data-crawler
cotoyogi
crawlspace
diffbot
duckassistbot
facebookbot
factset_spyderbot
firecrawlagent
friendlycrawler
google-cloudvertexbot
google-extended
googleother
googleother-image
googleother-video
gptbot
iaskspider/2.0
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
meta-externalagent
meta-externalagent
meta-externalfetcher
meta-externalfetcher
mistralai-user/1.0
novaact
oai-searchbot
omgili
omgilibot
operator
pangubot
panscient
panscient.com
perplexity-user
perplexitybot
petalbot
phindbot
qualifiedbot
quillbot
quillbot.com
sbintuitionsbot
scrapy
semrushbot-ocob
semrushbot-swa
sidetrade indexer bot
tiktokspider
timpibot
velenpublicwebcrawler
webzio-extended
wpbot
yandexadditional
yandexadditionalbot
youbot

aitcsroboti
accoona
admantx
admantx-usaspb
adbeat_bot
aihitbot
amazonbot
arachnophilia
aspiegelbot
awariosmartbot
backdoorbot
backrub
baidu
blexbot
blexbot
becomebot
blowfishi
bomborabot
catchbot
ccbot
cherrypicker
claudebot
clickagy
cliqzbot
coccocbot
converacrawler
contxbot
crowdtanglebot
cyberspyder
dotbot
echoboxbot
emailcollector
exabot
eyeotabot
findlinks
foobot
genieo
geturl
gigabot
grapeshotcrawler
gumgum
httrack
huaweisymantecspider
iascrawler
imagesiftbot
jikespider
jobboerse
java
jyxobot
leikibot
linkscan
linkisbot
linkdexbot
linkfluence.com
livelapbot
mail.ru_bot
mauibot
mazbot
mbcrawler
megaindex.ru
mj12bot
mojeekbot
mtbot/1.1.0i
nerdybot
nimbostratus-bot
ntentbot
offline explorer
onespot-scraperbot
openbot
outclicksbot
paperlibot
perl
petalbot
plurkbot
proximic
proximi
python
quantcastboti
qwantify
scholarbot
scrap
screaming frog seo spider
semantici
sentibot
seokicks
seokicks-robot
serendeputybot
serpstatbot
seznambot
sitecheck-sitecrawl
sitesnagger
snooper
sogou
sosospider
superbot
taboolabot
teleportpro
tkbot
ttd-content
tweetmemebot
urlspiderpro
vagabondo
velenpublicwebcrawler
voilabot
voluumdsp-content-bot
webcopier
weborama-fetcher
webreaper
webstripper
webzip
xaldon_webspider
yak
yandex
yandexbot
yandeximages
zgrab
zoominfobot
scrapy
buck
tinytestbot
semrushbot
ahrefsbot
petalbot
mj12bot
dotbot
mauibot
yandexbot
baiduspider
barkrowler
bytespider
whatstuffwherebot
applebot
sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
user-agent: sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou-test-spider/4.0 (compatible; msie 5.5; windows 98)
mozilla/5.0 (compatible; konqueror/3.5; linux) khtml/3.5.5 (like gecko) (exabot-thumbnails)
mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)
swiftbot
slurp
ccbot/2.0 (https://commoncrawl.org/faq/)
ccbot/2.0
ccbot/2.0 (http://commoncrawl.org/faq/)

googlebot
bingbot
duckduckbot

Comments

• START BOOKSWARM ROBOTS.TXT TEMPLATE
• -----------------------------------
• Set the crawl delay to 7 seconds - not all search engines will honour this
• Tell all user agents to ignore wp-admin
• Tell all user agents to ignore URLs with querystrings
• Block bots including AI bots
• Block these other bots
• Allow Googlebot and other specific bots
• ---------------------------------
• END BOOKSWARM ROBOTS.TXT TEMPLATE
• _
• [ ]
• ( )
• |>|
• __/===\__
• //| o=o |\\
• <] | o=o | [>
• \=====/
• / / | \ \
• <_________>