bugnard.ch
robots.txt

Groups

archive.org_bot
arquivo-web-crawler
heritrix
ia_archiver
ia_archiver-web.archive.org
nicecrawler

ahrefsbot
semrushbot
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
splitsignalbot
semrushbot-coub
barkrowler
blexbot
brightedge crawler
cocolyzebot
dataforseobot
domainstatsbot
dotbot
hypestat
linkdexbot
mj12bot
online-webceo-bot
screaming frog seo spider
senutobot
seobilitybot
seokicks
seolizer
serpstatbot
sitecheckerbotcrawler
zoombot

a patent crawler
adstxtcrawler
awesomecrawler
backlinkcrawler
birdcrawlerbot
cispa webcrawler
companybook-crawler
content crawler spider
converacrawler
crawler4j
crawlyprojectcrawler
deepcrawl
domaincrawler
e.ventures investment crawler
erocrawler
fast enterprise crawler
fast-webcrawler
fr-crawler
garlikcrawler
gingercrawler
gluten free crawler
grapeshotcrawler
gsa-crawler
ias crawler
icc-crawler
ip-web-crawler.com
it2media-domain-crawler
kbcrawl
lssrocketcrawler
mbcrawler
minicrawler
msiecrawler
netestate ne crawler
neticle crawler
nimblecrawler
peer39_crawler
rukicrawler
safesearch microdata crawler
seekport crawler
simplecrawler
sistrix crawler
tocrawl
tombapublicwebcrawler
trendkite-akashic-crawler
ubicrawler
tombapublicwebcrawler
usinenouvellecrawler
webcompanycrawler

url_spider_pro
toutiaospider
sosospider
sogou spider2
sogou inst spider
lnspiderguy
linespider
lb-spider
landau-media-spider
kenjin spider
k2spider
jikespider
jamie's spider
gnam gnam spider
baiduspider-video
baiduspider-news
baiduspider-image
etaospider
sogou spider

bazqux
bitlybot
bublupbot
embedly
flipboardproxy
freshrss
friendica
hatena
iframely
inoreader
mail.ru_bot
miniflux
newsblur
nextcloud
pocketparser
serendeputybot
simplepie
slackbot-linkexpanding
snap url preview service
startmebot
superfeedr
surdotlybot
synapse
tiny tiny rss
vkshare

bytespider
ccbot
claudebot
diffbot
gptbot
meta-externalagent
omgili
timpibot

dataprovider.com
dcrawl
httrack
httrack 3.0
metainspector
newspaper
nutch
offline explorer
openindexspider
scrapy

adbeat_bot
aihitbot
anderspinkbot
archivebot
awariobot
awariosmartbot
bitsightbot
blackboard
brandverity
cincraw
ev-crawler
hubspot
imagesiftbot
ioncrawl
jugendschutzprogramm-crawler
kstandbot
lightspeedsystemscrawler
linkfluence
linkwalker
magpie-crawler
mediatoolkitbot
muckrack
netcraftsurveyagent
netvibes
pandalytics
panscient.com
proximic
scoop.it
seekportbot
smtbot
trendictionbot
trendsmapresolver
turnitin
turnitinbot
tweetmemebot
twingly
um-ln
velenpublicwebcrawler
zoominfobot

eventmachine httpclient
niki-bot
istellabot
go-http-client/1.1
adscanner
ioncrawl
geedoproductsearch
pagepeeker
uptimebot.org
uptimerobot
coccoc
coccocbot
coccocbot-web

*

Other Records

Comments

• ARCHIVERS
• CRAWLERS
• UNCATEGORIZED CRAWLERS
• UNCATEGORIZED SPIDERS
• FETCHERS
• User-agent: Discordbot
• User-agent: facebookexternalhit
• User-agent: Feedly
• User-agent: Google Web Preview
• User-agent: Google-Read-Aloud
• User-agent: LinkedInBot
• User-agent: Mastodon
• User-agent: Pinterestbot
• User-agent: redditbot
• User-agent: SkypeUriPreview
• User-agent: snapchat
• User-agent: TelegramBot
• User-agent: Twitterbot
• User-agent: Viber
• User-agent: WhatsApp
• User-agent: Yahoo Link Preview
• AI SCRAPERS
• User-agent: Applebot-Extended
• User-agent: FacebookBot
• User-agent: Google-Extended
• SCRAPERS
• INTELLIGENCE GATHERERS
• User-agent: AdsBot-Google
• User-agent: AdsBot-Google-Mobile
• User-agent: Google-Safety
• User-agent: Mediapartners-Google
• +++
• BUGNARD

Warnings

• 4 invalid lines.