olightstore.com
robots.txt

Groups

*

googlebot-image

a1 sitemap generator
a1 website download
a6-indexer
aasp
abachobot
abonti
abotemailsearch
aboundex
aboutusbot
accmonitor compliance server
accoon
achulkov.net page walker
acme.spider
acoonbot
acquia-crawler
activetouristbot
acunetix
ad muncher
adamm bot
adbeat_bot
admantx
adminshop.com
advanced email extractor
aesop_com_spiderman
aespider
af knowledge now verity spider
afd-verbotsverfahren
aggregator:vocus
ah-ha.com crawler
aibot
aidu
aihitbot
aipbot
aisiid
aitcsrobot/1.1
ajsitemap
akamai-sitesnapshot
alexawebsearchplatform
alexfdownload
alexibot
alkalinebot
all acronyms bot
alligator
allsubmitter
almaden
alpha search agent
alphabot
alphaseobot
alphaseobot-sa
amazonbot
amerla search bot
amfibibot
ampmppc.com
amznkassocbot
anarchie
anemone
anonymous
anotherbot
answerbot
answerbus
answerchase prove
antbot
antibot
antisantyworm
antro.net
aonde-spider
apexoo
aport
appid: s~stremor-crawler-
applebot
aqua_products
arabot
arachmo
arachnophilia
archive
aria equalizer
arianna.libero.it
arikus_spider
artavisbot
artera
art-online.com
asaha search engine turkey
ask
aspider
aspiegelbot
aspseek
asterias
astrofind
athenusbot
atlocalbot
atomic_email_hunter
attach
attrakt
attributor
augurfind
auresys
autobaron crawler
autoemailspider
autowebdir
avsearch-
awariosmartbot
axfeedsbot
axios
axmo
axonize-bot
ayna
b2w
babbar
backdoorbot
backdoorbot/1.0
backlink-ceck
backlink-check
backlinkcrawler
backrub
backstreet
backstreet browser
backweb
badass
bandit
barkrowler
basalsa
batchftp
battleztar bazinga
baypup
bbbike
bbot
bcahibot
bdcbot
bdfetch
becomebot
becomejpbot
beetlebot
bender
besserscheitern-crawl
betabot
big brother
big data
bigado.com
bigcliquebot
bigfoot
biglotron
bilbo
bilgibetabot
bilgibot
bintellibot
bitacle
bitlybot
bitvouseragent
bizbot
bizworks retriever
black hole
black.hole
blackbird
blackboard
blackboard safeassign
blackmask.net search engine
blackwidow
bladder fusion
blaiz-bee
blexbot
blinkx
blitzbot
blog conversation project
blogmyway
blogpulselive
blogrefsbot
blogscope
blogslive
bloobybot
blow
blowfish
blowfish/1.0
blp_bbot
blp_bbot/0.1
blt
bluechipbacklinks
bnf.fr_bot
boaconstrictor
boardreader
boi_crawl_00
boia.org
boia-scan-agent
boitho
bolt
booch
bookmark buddy bookmark checker
bookmark search tool
bosug
bot apoena
botalot
botrighthere
botswana
bottybot
bpbot
braintime_search
brandprotect
brandwatch
brokenlinkcheck.com
browseremulator
browsermob
bruinbot
bsalsa
bsearchr&d
bspider
btbot
btsearch
bubing
buddy
buibui
buildcms crawler
builtbottough
builtwith
bullseye
bullseye/1.0
bumblebee
bunnyslippers
buscadorclarin
buscaplus robi
butterfly
buyhawaiibot
buzzbot
buzzsumo
byindia
byspider
byteserver
bytespider
bzbot
cacheblaster
caddbot
cafi
calculon
camcrawler
camelstampede
canon-webrecord
careerbot
cataguru
catchbot
catexplorador
cazoodle
cazoodlebot
ccbot
ccgcrawl
ccubee
cd-preload
cegbfeieh
ce-preload
cerberian drtrs
cert figleafbot
cfetch
cfnetwork
chameleon
charlotte
chatgpt-user
check&get
checkbot
checklinks
cheesebot
chemiede-nodebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chilkat
chinaclaw
chlooe
chromeframe
cipinetbot
cis455crawler
citeseerxbot
cizilla
clariabot
claritybot
climate ark
climateark spider
cliqzbot
cloud mapping
clshttp
clushbot
coast scan engine
coast webmaster pro
coccoc
coccocbot-web
cogentbot
cognitiveseo
collapsarweb
collector
colocrossing
com.plumanalytics
combine
connectsearch
conpilot
contentsmartz
contextad bot
contype
cookienet
coolbot
coolcheck
copernic
copier
copyrightcheck
copyscape
core-project
cosmos
covario-ids
cowbot-
cowdog bot
crabbybot
craftbot
crawl.sogou.com
crawl_application
crawl3r
crawler.feedback
crawler.kpricorn.org
crawler@
crawler_for_infomine
crawler43.ejupiter.com
crawler4j
crawly
crazywebcrawler
creativecommons
crescent
crescent internet toolpak http ole control v.1.0
crmnlcrawlagent
cs-crawler
cse html validator
cshttp
cshttpclient
cuasarbot
culsearch
curious
curl
custo
cvaulev
cyberdog
cybernavi_webget
cyberpatrol sitecat webbot
cyberspyder
cydralspider
d1garabicengine
danibot
databasedrivermysqli
datacha0s
dataforseobot
datafountains
dataparksearch
dataprovider.com
datascape robot
dataspearspiderbot
dataspider
dattatec.com
daum.net
daumoa
dblbot
dcpbot
declumbot
deepindex
deepnet crawler
deeptrawl
dejan
del.icio.us-thumbnails
deltascan
delvubot
demandbase-bot
demon
deusu
devil
dfs-fetch
diagem
diamond
dibot
didaxusbot
digext
digger
digincore
digi-rssbot
digitalarchivesbot
digitalpebble
digout4u
diibot
dillo
dir_snatch.exe
dirbuster
disco
discobot
discoverybot
distilled-reputation-monitor
dittospyder
djangotraineebot
dkimrepbot
dmoz downloader
dnyzbot
doc
docomo
dof-verify
domainappender
domaincrawler
domainscan
domainsigmacrawler
domainstatsbot
domainwatcher bot
dotbot
dotspotsbot
dow jones searchbot
download
download wonder
downloader
doy
dragonfly
drip
drone
dtaagent
dts agent
dtsearchspider
dumbot
dwaar
dxseeker
eah
earth platform indexer
earth science educator robot
easouspider
easydl
ebingbong
ec2linkfinder
ecairn-grabber
ecatch
eccp/1.0
echoosebot
ecosearch
ecxi
edisterbot
edugovsearch
egothor
eidetica.com
eirgrabber
elblindo the blind bot
elisabot
ellerdalebot
email extractor
email siphon
email wolf
emailcollector
emailleach
emailsiphon
emailwolf
emeraldshield
empas_robot
enabot
endeca
enigmabot
enswer neuro bot
enter user-agent
entitycubebot
erocrawler
e-societyrobot
estylesearch
esyndicat bot
eurosoft-bot
evaal
evc-batch
eventware
everest-vulcan inc.
evil
exabot
exactsearch
exactseek
exooba
expired domain sleuth
exploder
express webpictures
extlinksbot
extractor
extractorpro
extreme picture finder
eyemon
eyenetie
ezooms
ez-robot
factbot
fairad client
falcon
fast data search document retriever
fast esp
fastbot crawler
fastbot.de crawler
fast-search-engine
fatbot
favcollector
faviconizer
favorites sweeper
f-bot test pilot
fdm
fdse robot
fedcontractorbot
fembot
femtosearchbot
fetch api request
fetch_ici
fgcrawler
fhscan
filangy
filehound
fimap
findanisp.com_isp_finder
findlinks
findweb
firebat
firefox/7.0
firstgov.gov search
flaming attackbot
flamingo_searchengine
flashcapture
flashget
flickysearchbot
fluffy the spider
flunky
focused_crawler
followsite
foobot
fooooo_web_video_crawl
fopper
formulafinderbot
forschungsportal
fq
fr_crawler
francis
franklin locator
freeuploader
freewebmonitoring sitechecker
freshcrawler
freshdownload
freshlinks.exe
friendfeedbot
frodo.at
froggle
frontpage
froola bot
full_breadth_crawler
fu-nbi
funnelback
furlbot
fyberspider
fyrebot
g10-bot
gaisbot
galaxybot
garlik
garlikcrawler
gazz
gbplugin
generate_infomine_category_classifiers
genevabot
geniebot
genieo
geomaxenginebot
geometabot
geonabot
geovisu
germcrawler
gethtmlcontents
getintent
getleft
getright
getsmart
geturl.rexx
getweb
giant
gigablast
gigablastopensource
gigabot
g-i-g-a-b-o-t
girafabot
gleamebot
gnome-vfs
go!zilla
go-ahead-got-it
goforit.com
goforitbot
go-http-client
gold crawler
goldfire server
golem
goodjelly
google-extended
gordon-college-google-mini
goroam
goseebot
gotit
govbot
gozilla
gptbot
gpu p2p crawler
grabber
grabnet
grafula
grapefx
grapeshot
grapeshotcrawler
grbot
greenyogi
gridbot
gromit
grub
gslfbot
gt::www
gulliver
gulperbot
gurujibot
gvc business crawler
gvc crawler
gvc search bot
gvc web crawler
gvc weblink crawler
gvc world links
gvcbot.com
gwene
haansoft
haosouspider
happyfunbot
harvest
harvest/1.5
hatena
hatena antenna
havij
hawler
hcat
hclsreport-crawler
hd nutch agent
header_test_client
headmasterseo
healia
helix
here will be link to crawler site
heritrix
hiscan
hisoftware accmonitor server
hisoftware accverify
hitcrawler
hivabot
hloader
hmsebot
hmview
hoge
holmes
homepagesearch
hooblybot-image
hoowwwer
hostcrawler
hsft - link scanner
hsft - lvu scanner
hslide
ht://check
htdig
html link validator
htmlparser
http::lite
httplib
httrack
huaweisymantecspider
hubspot crawler
hubspotcontentsearchbot
hughcrawler
hul-wax
humanlinks
hybridbot
hypefactors
hyperestraier
hyperix
ia_archiver
iaea.org
iblog
ibuena
icab
icds-ingestion
ichiro
iconsurf
icopyright conductor
idbot
idg/uk
id-search
ieautodiscovery
iecheck
ihwebchecker
iiitbot
iim_405
ilsebot
iltrovatore
iltrovatore-setaccio
image fetch
image stripper
image sucker
imagebot
image-fetcher
imagefortress
imageshereimagesthereimageseverywhere
imagevisu
imds_monitor
imo-google-robot-intelink
inagist.com url crawler
indeedbot
indexer
industry cortex webcrawler
indy library
indylabs_marius
inelabot
inet32 ctrl
inetbot
info seeker
infolink
infomine
infonavirobot
informant
infoseek sidewinder
infotekies
infousabot
ingrid
inktomi
insightscollector
insightsworksbot
inspirebot
instabid
insumascout
intelix
intelliseek
interget
internet ninja
internet radio crawler
internetlinkagent
internetseer
internetvista monitor
interseek
ioi
ipadd bot
ips-agent
ipselonbot
ip-web-crawler.com
iria
irlbot
iron33
isara
isearch
isilox
iskanie
istellabot
its-learning crawler
iu_csci_b659_class_crawler
iupui
ivia
jadynave
jamesbot
jbot
jbrofuzz
jemmathetourist
jennybot
jersey
jetbot
jetbrains omea pro
jetcar
jikespider
jim
jobo
jobspider_ba
joc
joc web spider
joedog
joomla
jorgee
joyscapebot
jspyda
junut bot
justview
jyxobot
k.s.bot
kakclebot
kalooga
katatudo-spider
kbeta1
keepni web site monitor
kenjin spider
kenjin.spider
keybot translation-search-machine
keywenbot
keyword density
keyword density/0.9
keyword.density
kinjabot
kitenga-crawler-bot
kittiecentral
kiwistatus
kmbot-
kmccrew bot search
knight
knowitall
knowledge engine
knowledge.com
koepabot
koninklijke
korniki
kozmosbot
krowler
ksbot
kuloko-bot
kulturarw3
kummhttp
kurzor
kyluka crawl
l.webis
labhoo
labourunions411
lachesis
lament
lamerexterminator
lanshanbot
lapozzbot
larbin
lbot
lcabotaccept
leaptag
leechftp
leechget
letscrawl.com
lexibot
lexxebot
lftp
libcrawl
libiviacore
libw
libweb
libweb/clshttp
libwhisker
liebaofast
lightspeedsystems
likse
linespider
linguee
linguee bot
link checker
link validator
link_checker
linkalarm
linkbot
linkcheck by siteimprove.com
linkcheck scanner
linkchecker
linkdex.com
linkdexbot
linkextractorpro
linkfluence
linklint
linklooker
linkman
linkpad
linkpadbot
links sql
linkscan
linkscan/8.1a unix
linksmanager
linksmanager.com_bot
linksweeper
linkwalker
linqiametadatadownloaderbot
linqiarssbot
linqiascrapebot
lipperhey
litefinder
litemage_walker
litlrbot
little grabber at skanktale.com
livelapbot
lm harvester
lmqueuebot
lmspider
lnspiderguy
loadtimebot
localcombot
locust
lolongbot
lookbot
lsearch
lssbot
lt scotland checklink
ltx71
lwp
lwp::simple
lwp-request
lwp-trivial
lwp-trivial/1.34
lycos_spider
lydia entity spider
lynnbot
lytranslate
magnet
mag-net
magpie
magpie-crawler
magus bot
mail.ru
mail.ru_bot
mainseek_bot
majestic12
mammoth
map robot
markmonitor
markwatch
masagool
masidani_bot_
mass downloader
masscan
mata hari
mata.hari
matentzn at cs dot man dot ac dot uk
mauibot
maxamine.com
maxomobot
mbcrawler
mcbot
meanpathbot
mediawords
medrabbit
megaindex.ru
megaindex.ru/2.0
megite
memacbot
memo
mendeleybot
mercator-
mercuryboard_user_agent_sql_injection.nasl
metacarta
metaeuro web search
metager2
metagloss
metal crawler
metaquerier
metaspider
metaspinner
metatagrobot
metauri
mfc_tear_sample
mfcrawler
mfhttpscan
microsoft data access
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
midown tool
miixpc
miixpc/4.2
minibot
minirank
mini-robot
mirror
missigua locator
mister pix
mister.pix
miva
mj12bot
mnogosearch
mod_accessibility
moduna.com
moget
moget/2.1
mojeek
mojeekbot
monkeycrawl
morfeus fucking scanner
moses
mowserbot
mozdex
mqbot
mqqbrowser
mr.4x3
ms web services client protocol
mse360
msfrontpage
msiecrawler
msindianwebcrawl
msmobot
msnptc
msrabot
msrbot
muhstik-scan
musobot
name intelligence
nameprotect
naver
navroad
nearsite
needle
nessus
net vampire
netants
netcraft
netestate ne crawler
netlyzer
netmechanic
netresearchserver
netspider
nettrack
netvibes
netzip
nextgensearch
nextgensearchbot
nibbler
nicerspro
niki-bot
nikto
nimblecrawler
nimbostratus-bot/v1.3.2
ninja
nmap
npbot
nutch
nyu
objectssearch
obot
octopus
offline explorer
offline navigator
omgili
openfind
openfind data gathere
openlinkprofiler
opennet.ru
openvas
oracle ultra search
orangebot
orangespider
outclicksbot
outfoxbot
page analyzer
page scorer
pageanalyzer
pagegrabber
pagescorer
panscient
papa foto
paperlibot
pavuk
pcbrowser
pecl::http
peerbot
peoplepal
petalbot
phpcrawl
picofeed
picscout
picsearch
picturefinder
pictureofinternet
pimonster
pi-monster
piplbot
pixray
plantynet
pleasecrawl
plumanalytics
pockey
poe-component-client-http
probethenet
propowerbot
propowerbot/2.14
prowebwalker
psbot
pump
pycurl
python
quepasacreep
queryn metasearch
queryseekerspider
qwantify
qwant-news
rankactive
rankactivelinkbot
rankflex
rankingbot
rankingbot2
rankivabot
rankurbot
realdownload
reaper
rebelmouse
recorder
redesscrapy
reget
repomonkey
repomonkey bait & tackle/v1.01
ripper
rma
rocketcrawler
rogerbot
rytebot
salesintelligent
sbider
scan.lol
scanalert
scanbot
scrapy
screaming
screaming frog seo spider
screenerbot
scspider
searchestate
searchmetricsbot
seekport
seekport crawler
semrush
semrushbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot-si
semrushbot-swa
sentibot
seokicks
seolyticscrawler
seomoz
seoprofiler
seoscanners
seostats
serendeputybot
serpstatbot
sexsearcher
seznam
seznambot
shodan
siphon
sistrix
site sucker
siteauditbot
sitebeam
sitecheck.internetseer.com
siteexplorer
siteimprove
sitelockspider
sitesnagger
sitesucker
sitevigil
slackbot-linkexpanding
slurp
slysearch
smartdownload
smtbot
snake
snapbot
snoopy
socialrankiobot
soft411
sogou
sogou web spider
sogouspider
sosospider
sottopop
spacebison
spammen
spankbot
spanner
spbot
spider.acont.de
spinn3r
splitsignalbot
sputnikbot
sputnikbot/2.3
sqlmap
sqlworm
sqworm
ssm agent
steeler
stripper
studio
sucker
sucuri
superbot
superhttp
surfbot
surveybot
suzuran
swiftbot
sysscan
szukacz
szukacz/1.4
t0phackteam
t8abot
takeout
tamu
teleport
teleportpro
telesoft
telesphoreo
telesphorep
the intraformant
the knowledge ai
thenomad
theusefulbot
tighttwatbot
timpibot/0.8
tinytestbot
titan
tkbot
toata
tocrawl/urldispatcher
toweyabot
trendiction
trendiction.com
trendiction.de
trendictionbot
true_robot
true_robot/1.0
turingos
turnitin
turnitinbot
tutorgig
tutorial crawler
twengabot
twice
typhoeus
ubicrawler
ucbrowser
unisterbot
urly warning
urly.warning
vacuum
vagabondo
vb project
vci
vci webviewer vci webviewer win32
vericitecrawler
vidiblescraper
virusdie
voideye
voil
voltron
wadoo
wallpapers/3.0
wallpapershd
wasalive-bot
wbsearchbot
web auto
web collage
web enhancer
web fetch
web fuck
web image collector
web pix
web sauger
web sucker
webalta
webauto
webbandit
webbandit/3.50
webcollage
webcopier
webdav
webenhancer
webfetch
webfuck
webgo is
webimagecollector
webleacher
webmasterworldforumbot
webmeup-crawler
weborama
webpix
webreaper
websauger
website quester
websiteextractor
websitequester
webster
webster pro
webstripper
websucker
webwhacker
webzip
webzip/4.0
wesee
wget
wget/1.5.3
wget/1.6
whack
whacker
whatweb
who.is bot
widow
winhttrack
wiseguys robot
wisenutbot
wonderbot
woobot
wotbox
wprecon
wpscan
www::mechanize
www-collector-e
www-mechanize
wwwoffle
x09mozilla
x22mozilla
xaldon webspider
xaldon_webspider
xenu
xenu's
xenu's link sleuth 1.1c
xpymep1.exe
yacybot
yeti
y!j-bru/vsidx
y!j-brw
y!j-asr
y!j-mmp/dscv
y!j-wsc
y!j-dlc
yj-bru/vsidx
yj-brw
yj-asr
yj-mmp/dscv
yj-wsc
yj-dlc
yisouspider
youdaobot
zade
zao
zauba
zauba.io
zermelo
zeus
zeus 32297 webster pro v2.9 win32
zgrab
zhanzhang
zipppbot
zitebot
zmeu
zoominfobot
zumbot
zyborg

Other Records

Comments

• Robots.txt file for www.olightstore.com
• Allow search crawlers to discover the sitemap
• For all robots
• CSS, JS, Images, Files
• Block access to specific groups of pages
• Tracking Parameters
• Search results
• Similar content
• Dynamic parameter duplicate URL
• Member page
• Invalid page
• Good bots
• Ban bots that don't benefit us
• Bad bot Count: 1345
• Editor: Paul Zhang
• Last updated: 6/7/2024

Warnings

• 13 invalid lines.