stocklight.com
robots.txt

Groups

baiduspider
googlebot
googlebot-mobile
googlebot-image
googlebot-news
googlebot-video
adsbot-google
adsbot-google-mobile
feedfetcher-google
mediapartners-google
mediapartners
apis-google
bingbot
slurp
duckduckbot
linkedinbot
twitterbot
whatsapp
chrome-lighthouse
duckduckgo-favicons-bot
google-adwords-instant
appengine-google
google web preview
google-structured-data-testing-tool
google-physicalweb
google favicon
google-site-verification
amazon cloudfront
w3c_validator
w3c-checklink
w3c-mobileok
w3c_i18n-checker
w3c_css_validator
w3c_unicorn

amazonbot

Other Records

a6-indexer
ahc
aisearchbot
ahrefs
ahrefsbot
alphabot
amazonbot
anyevent
apache-httpclient
addsearchbot
adidxbot
advbot
adbeat_bot
adobe web capture
archive.org_bot
archivebot
aspiegelbot
awariorssbot
awariosmartbot
b2b bot
bdcbot
biglotron
blp_bbot
baidu-yunguance
bark[rr]owler
bazqux
behloolbot
bingbot
bingpreview
blackboard
bomborabot
brandonbot
brandverity
buck
bublupbot
bubing
bitbot
bitlybot
brandverity
blexbot
ccbot
cc metadata scaper
caliperbot
capsulechecker
careerbot
cliqzbot
cloudflare-alwaysonline
collection@infegy.com
companybook-crawler
convera
contextad bot
crunchbot
crystalsemanticsbot
cincraw
cyberpatrol
daum
dcbot
deusu
diffbot
disqus
domain re-animator bot
domaincrawler
domains project
dotbot
ec2linkfinder
ecxios
embedly
eright
everyonesocialbot
exabot
extlinksbot
ezid
eyeson
fast-webcrawler
fast enterprise crawler
feed validator
feedly
feedspot
femtosearchbot
finditanswersbot
findxbot
findlink
flamingo_searchengine
fyrebot
g2 web services
g2reader-bot
g00g1e.net
gbot
gigablast
gigabot
gigabot
gingercrawler
gnam gnam spider
go-http-client
gnu-wget
gnowitnewsbot
google-xrawler
grapeshotcrawler
grub.org
grobbot
hbtop
httrack
hubspot
ia_archiver
ias crawler
icbot
ichiro
infoobot
infoseek
indeedbot
interfaxscanbot
ip-web-crawler.com
ips-agent
istellabot
it2media-domain-crawler
jamie's spider
jobboersebot
jugendschutzprogramm-crawler
jyxobot
k7mlwcbot
kemvibot
kosmiobot
lb-spider
lcc
libwww-perl
linguee bot
linkarchiver
linkisbot
mappydata
mail.ru_bot
magpie-crawler
mastodon
meltwaternews
metajobbot
metauri
mlbot
msnbot
msrbot
mozilla/5.0
muckrack
nerdbynature.bot
netcraftsurveyagent
netestate ne crawler
netresearchserver
niki-bot
nimbostratus-bot
ning
nutch
nutch
nutchcvs
nutchorg
nuzzel
nutch
ocarinabot
okhttp
online-webceo-bot
openhosebot
openindexspider
outbrain
outclicksbot
p2p-crawler
panscient
paperlibot
page2rss
pagepeeker
pagesinventory
papago
paulbot
pbmservus
php/5.2.9
phpcrawl
pingdom
pocketparser
polyshock
pr-cy.ru
privacyawarebot
primalbot
protopage
proximic
proximic
purebot
python-urllib
python-requests
qbik
qirina crawler
quicklook
quora
rainbowlink_compatibility_verifier
rankactivelinkbot
rbbot
redditbot
refindbot
retrevopageanalyzer
revoseekbot
robot
rogerbot
safednsbot
safetybis
saucerbot
sb-search
scanscout
scrapy
screaming frog seo spider
search.yahoo.com
searchmetricsbot
seekbot
seekport crawler
semrushbot
semanticbot
semanticscholarbot
sentibot
serpstatbot
sgbot
shareaholicbot
sharkrflinkbot
sitebot
siteexplorer.info
sistrix crawler
simplecrawler
simplereach
simplescraper
skimbot
skypeuripreview
slack-imgproxy
slackbot
slurp
smooci
smtbot
snacktory
socialrankiobot
somero
sosoimagespider
sosospider
speedy
spir
spottobot
sputnikbot
s2crawl
surfbot
synoobot
system-center-operations-manager
taboolabot
tagoobot
teoma
tineye
tinderbot
tineye
toutiaospider
trendictionbot
tumblr
twitterbot
twengabot
twingly
turnitinbot
tweetedtimes
tweetmemebot
twurly
twbot
uptimerobot.org
urlappendbot
urlfan-bot
vebidoo
vebidoobot
veloscope
verificalbot
versioneye-bot
vincibot
vkbot
vkshare
vocusbot
voilabot
w3c checker
w3c css validation service
w3c_validator
w3cchecklink
wappalyzer
wbsearchbot
websanscraper
webcompanycrawler
webcopier
webmon
website-monitoring
wesee:search
wget
whalebonebot
whakaoko-bot
whatweb
whatsmyip.org
whydisbot
winhttp
woobot
woobot
worio
woriobot
wotbox
wp engine system
wpscan
www-mechanize
www::mechanize
yandexmetrika
yandexturbo
yandeximageresizer
yandexvideoparser

*

Other Records

Warnings

• 3 invalid lines.