karl.com
robots.txt

Groups

*

a1 sitemap generator
abachobot
abcdatos botlink
aboundexbot
aboutusbot
accoona-ai-agent
addsugarspiderbot
adidxbot
ahoy! the homepage finder
ahrefsbot
aitcsrobot/1.1
alexibot
amznkassocbot
aqua_products
arachnophilia
architextspider
aspider/0.09
asterias
auresys/1.0
awariorssbot
awariosmartbot
b2w/0.1
backdoorbot/1.0
backrub/.
baiduspider
baiduspider-image
baiduspider-news
baiduspider-video
becomebot
beslistbot
big brother
bizbot003
black hole
blackwidow
blexbot
blowfish
blowfish/1.0
blp_bbot
bookmark search tool
botalot
botrighthere
builtbottough
bullseye/1.0
bunnyslippers
businessdbbot
cactvs chemistry spider
catchbot
ccbot
cegbfeieh
checkbot/x.xx lwp/5.x
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
coccoc
combine/0.0
conceptbot/0.3
converacrawler
copernic
copyrightcheck
cosmos
covarioids
crescent
crescent internet toolpak http ole control v.1.0
custo
cyberpatrol sitecat webbot
cyberspyder/2.1
daumoa
deweb/1.01
disco
disco pump 3.0
disco pump 3.2
discobot
discofinder
dittospyder
dotbot
download demon
download demon/3.2.0.8
download demon/3.5.0.11
download ninja
dumbot
ecatch
ecatch/3.0
eirgrabber
email exractor
emailcollector
emailsiphon
emailwolf
enigmabot
envolk
erocrawler
exabot
explorersearch
express webpictures
extractorpro
eyenetie
ezooms
fairad client
fdm 3.x
fdse robot
felixide/1.0
fido/0.9 harvest/1.4.pl2
fish-search-robot
flaming attackbot
flashget
flashget webwasher 3.2
flaxcrawler
foobot
freecrawl
frontpage
gaisbot
genieo
getright/4.2
getweb!
gigabot
girafabot
go!zilla
golem/1.1
grabber
grabnet
grafula
grapeshot
gromit/1.0
grub
grub-client
gsa-crawler
gslfbot
happyfunbot
harvest/1.5
hatena antenna
hazel's ferret web hopper
heritrix
hloader
hmview
httplib
httrack
huaweisymantecspider
humanlinks
ia_archiver
image stripper
image sucker
inagist.com url crawler
incywincy/1.0b1
indy library
infonavirobot
informant
ingrid/0.1
intelium_bot
interget
internet ninja 6.0
irlbot
iron33/1.0.2
israelisearch/1.0
istellabot
iti spider
jennybot
jetcar
joc web spider
jubiirobot
jumpstation
katipo/1.0
kenjin spider
keyword density/0.9
labelgrab/1.1
larbin
leechftp
lemurwebcrawler
lexibot
libweb/clshttp
libwww-perl
linguee bot
linkdexbot/2.1
linkedinbot
linkextractorpro
linklooker
linkscan/8.1a unix
linkwalker
lnspiderguy
looksmart
lwp-trivial
lwp-trivial/1.34
magpie-crawler
mass downloader/2.2
mata hari
mediafox/x.y
merzscope
metagopher
metamojicrawler
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
midown tool
miixpc
miixpc/4.2
mister pix
mnogosearch
moget
moget/2.1
momspider/1.00 libwww-perl/0.40
motor/0.2
mozilla/4.0 (compatible; bullseye; windows 95)
msiecrawler
msrbot
msr-isrccrawler
naver
naverbot
navroad
nearsite
neosciocrawler
net vampire/3.0
netants
netcarta cyberpilot pro
netmechanic
netscoop/1.0 libwww/5.0a
netspider
netzip
nhsewalker/3.0
nicerspro
nomad-v2.x
northstar
npbot
nutch
obot
occam/1.0
octopus
offline explorer
ogspider
omniexplorer_bot
open text site crawler v1.0
openacoon
openbot
openfind
openfind data gathere
oracle ultra search
pagegrabber
pagepeeker
papa foto
pavuk
pcbrowser
perman
pgp-ka/1.2
plukkie
propowerbot/2.14
prowebwalker
proximic
psbot
python-urllib
queryn metasearch
r6_commentreader
r6_feedfetcher
radiation retriever 1.1
realdownload/4.0.0.42
reget
repomonkey
resume robot
rma
roverbot
ruby
safetynet robot 0.1
scoutjet
searchpreview
senrigan/xxxxxx
seokicks
seznambot
sitesnagger
slysearch
smartdownload
snooper/b97_01
sogou web spider
solbot/1.0 lwp/5.07
sootle
sosospider
spankbot
spanner
spanner/1.0 (linux 2.0.27 i586)
spbot
speedy spider
spyder3.microsys.com
superbot
superhttp/1.0
surfbot
surveybot
suzuran
szukacz/1.4
takeout
teleport
teleportpro
telesoft
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
turnitinbot
twiceler
ucsd-crawler
unisterbot
unwindfetchor/1.0
url control
url_spider_pro
urlck/1.2.3
urly warning
valkyrie/1.0 libwww-perl/0.40
vbseo
vci
vci webviewer vci webviewer win32
voideye
voilabot
wbsearchbot
web image collector
web sucker
webauto
webbandit
webbandit/3.50
webcopier
webenhancer
webferret
webfetch
webleacher
weblexbot
webmasterworld extractor
webmasterworldforumbot
webreaper
websauger
website quester
webster pro
webstripper
webvac
webwalk
webwatch
webwhacker
webzip
webzip/4.0
wget
wget/1.5.3
wget/1.6
whowhere robot
widow
www-collector-e
xaldon webspider
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
zyborg

Other Records

Comments

• DEFAULT ROBOTS.TXT INSTRUCTIONS ON SFCC
• TO BE ACTIVATED DEPENDING ON FILTERS AND PRODUCT VARIATIONS
• Disallow: */?tiles
• Disallow: */?prefn1
• Disallow: */?cgid
• Disallow: */Product-Variation?
• Disable Spam User Agent

Warnings

• 5 invalid lines.