bergerholding.eu
robots.txt

Groups

aboundexbot
ahrefsbot
aihitbot
amazonbot
anthropic-ai
applebot
applebot-extended
archive.org_bot
backlinkcrawler
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cliqzbot
cohere-ai
dataprovider
diffbot
domaincrawler
dotbot
easouspider
ec2linkfinder
exabot
ezooms
facebookbot
facebookexternalhit
fetch
friendlycrawler
genieo
go-http-client/2.0
gptbot
grub-client
httrack
ia_archiver
ia_archiver/1.6
ia_archiver-web.archive.org
icc-crawler
imagesiftbot
img2dataset
infopath
infopath.2
ip-web-crawler.com
libwww
linkpadbot
mail.ru
meanpathbot
meta-externalagent
meta-externalfetcher
microsoft.url.control
mj12bot
mozilla/4.0
msiecrawler
netestate ne crawler
npbot
oai-searchbot
offline explorer
omgili
omgilibot
panscient.com
perplexitybot
psbot
scrapy
screaming frog seo spider
searchmetericsbot
searchspider
semrushbot
seokicks-robot
sitebot
sitecheck.internetseer.com
sitesnagger
sosospider
spbot
swebot
taptubot
teleport
teleportpro
timpibot
turnitinbot
twengabot
twiceler
ubicrawler
velenpublicwebcrawler
vscooter
wbsearchbot
webcapture
webcopier
webreaper
webstripper
webzip
wget
wotbox
xenu
xenu's
xenu's link sleuth 1.1c
yandex
youbot
zealbot

*

Other Records