*
| Rule |
Path |
| Disallow |
/app/ |
| Disallow |
/bin/ |
| Disallow |
/dev/ |
| Disallow |
/lib/ |
| Disallow |
/phpserver/ |
| Disallow |
/pkginfo/ |
| Disallow |
/report/ |
| Disallow |
/setup/ |
| Disallow |
/update/ |
| Disallow |
/var/ |
| Disallow |
/vendor/ |
| Disallow |
/catalog/product_compare/ |
| Disallow |
/catalog/category/view/ |
| Disallow |
/catalog/product/view/ |
| Disallow |
/catalogsearch/ |
| Disallow |
/search/ |
| Disallow |
/checkout/ |
| Disallow |
/control/ |
| Disallow |
/contacts/ |
| Disallow |
/customer/ |
| Disallow |
/customize/ |
| Disallow |
/newsletter/ |
| Disallow |
/review/ |
| Disallow |
/sendfriend/ |
| Disallow |
/wishlist/ |
| Disallow |
/stores/store/ |
| Disallow |
/catalog/seo_sitemap/ |
| Disallow |
/ajaxcart/ |
| Disallow |
/ajax/ |
| Disallow |
/pos/ |
| Disallow |
/sales/guest/form/ |
| Disallow |
/checkout/cart/ |
| Disallow |
/productalert/ |
| Disallow |
/stores/ |
| Disallow |
/stores/store/redirect/ |
| Disallow |
/stores/store/switch/ |
| Disallow |
/downloadable/ |
| Disallow |
/subscription/ |
| Disallow |
/newsletter/ |
| Disallow |
/push_notification/ |
| Disallow |
/brands.html |
| Disallow |
/marken.html |
| Disallow |
/merken.html |
| Disallow |
/marques.html |
| Disallow |
/privacy-policy-cookie-restriction-mode/ |
| Disallow |
/no-route/ |
| Disallow |
/enable-cookies/ |
| Disallow |
/rest/ |
| Disallow |
/composer.json |
| Disallow |
/composer.lock |
| Disallow |
/CONTRIBUTING.md |
| Disallow |
/CONTRIBUTOR_LICENSE_AGREEMENT.html |
| Disallow |
/COPYING.txt |
| Disallow |
/Gruntfile.js |
| Disallow |
/LICENSE.txt |
| Disallow |
/LICENSE_AFL.txt |
| Disallow |
/nginx.conf.sample |
| Disallow |
/package.json |
| Disallow |
/php.ini.sample |
| Disallow |
/RELEASE_NOTES.txt |
| Disallow |
/*?*product_list_mode= |
| Disallow |
/*?*product_list_order= |
| Disallow |
/*?*product_list_limit= |
| Disallow |
/*?*product_list_dir= |
| Disallow |
/*?SID= |
| Disallow |
/*? |
| Disallow |
/*.php$ |
| Allow |
/*?p=* |
| Disallow |
/*.CVS |
| Disallow |
/*.zip$ |
| Disallow |
/*.svn$ |
| Disallow |
/*.idea$ |
| Disallow |
/*.sql$ |
| Disallow |
/*.tgz$ |
addthis
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
alexabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
amznkassocbot/4.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baidu
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-image
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bdcbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bdcbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bsalsa
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
butterfly
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ccbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
changedetection
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
coccoc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
crazywebcrawler-spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
daumoa
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
docomo/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
easouspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
exabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ezooms
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
findxbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
findxbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
flipboard
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
genieo
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gluten free crawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
grapeshot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
haosouspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
hypercrawl
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ichiro
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ichiro/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ichiro/4.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
infoweb
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
istellabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
libwww-perl
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linguee bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linkdexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linkdexbot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linkdexbot/2.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linkdexbot/2.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
linkpadbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ltx71
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
meanpathbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex.ru
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex.ru/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
naverbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
nerdybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
netseer
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
nutch bots
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
obot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
paperlibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
phpservermon/3.1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
privacyawarebot/1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
proximic
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
psbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
queryseekerspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
qwantify
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
rogerbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
screenerbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
searchmetricsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot-sa
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sf/2 (skipfish)
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
siteexplorer
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sogou head spider/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sogou orion spider/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sogou pic agent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sogou pic spider/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sputnikbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sunrise
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
surdotlybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
surdotlybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
swiftbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
tdjbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trovitbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
turnitinbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
tweetmeme
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
tweetmemebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twengabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twengabot-2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twengabot-discover
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twengabot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
unwindfetchor
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
uptimebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
uptimebot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
uptimerobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
uptimerobot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
voltron
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
wbsearchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
wesee
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
wget
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
wget
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
wotbox
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
xovibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
y! j-asr
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yacybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yeti
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yodaobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yoozbot-2.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
youdaobot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yahoo! slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yahoo-slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
anews/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
anews
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
kribrum
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
kribrum/0.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
iskanie
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
paperlibot/2.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
special_archiver/3.1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
special_archiver
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ccbot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
exabot/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot/1.1~bl
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot/1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot-sa/0.97
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-fc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-fc/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-ln
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-ln/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
heritrix
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
deusu/5.0.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bitlybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bitlybot/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
maxpointcrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
getintent crawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
scrapy
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
scrapy/1.2.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twitterbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
twitterbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sputnikbot/2.3
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trident/6.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trident/5.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trident
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
guzzlehttp/6.1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
guzzlehttp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
x11
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
stackrambler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
metaspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
aport
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
scooter
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trendictionbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
trendictionbot0.5.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
kraken
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
kraken/0.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dotbot/1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bubing
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yeti/1.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
seznambot/3.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
webartexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
blexbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
spbot/5.0.3
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
spbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-ic
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
um-ic/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
germcrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
sogou
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gingercrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
nutch
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
urlbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
scoutjet
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
grouphigh
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
grouphigh/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.3
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.2.4
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.2.3
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.0.8
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.0.7
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.0.6
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.0.5
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.5
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.6
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.7
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot/v1.4.8
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
rogerbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
rogerbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex.ru
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
crawler4j
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
steeler/3.5
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
steeler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/5.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/5.2
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/2.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/3.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/4.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/5.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ahrefsbot/3.1
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
serpstatbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
boitho
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
ia_archiver
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
Comments