simonetta.it
robots.txt

Robots Exclusion Standard data for simonetta.it

Archived Snapshots

Resource Scan

Scan Details

Site Domain	simonetta.it
Base Domain	simonetta.it
Scan Status	Ok
Last Scan	2024-11-04T05:03:43+00:00
Next Scan	2024-12-04T05:03:43+00:00

Last Scan

Scanned	2024-11-04T05:03:43+00:00
URL	https://simonetta.it/robots.txt
Domain IPs	185.56.219.187, 93.63.231.174
Response IP	185.56.219.187
Found	Yes
Hash	63c61666e01eba107dfbe15f702c380d5703337cc128313a9ad88f7081bbc7dd
SimHash	7ad14b37e6b0

Groups

*

Rule	Path
Disallow	/CVS
Disallow	/*.svn$
Disallow	/*.idea$
Disallow	/*.sql$
Disallow	/*.tgz$
Disallow	/shop/admin/
Disallow	/shop/app/
Disallow	/shop/downloader/
Disallow	/shop/errors/
Disallow	/shop/includes/
Disallow	/shop/lib/
Disallow	/shop/pkginfo/
Disallow	/shop/shell/
Disallow	/shop/var/
Disallow	/shop/api.php
Disallow	/shop/cron.php
Disallow	/shop/cron.sh
Disallow	/shop/error_log
Disallow	/shop/get.php
Disallow	/shop/install.php
Disallow	/shop/LICENSE.html
Disallow	/shop/LICENSE.txt
Disallow	/shop/LICENSE_AFL.txt
Disallow	/shop/README.txt
Disallow	/shop/RELEASE_NOTES.txt
Disallow	/shop/?dir
Disallow	/shop/*?dir=desc
Disallow	/shop/*?dir=asc
Disallow	/shop/*?limit=all
Disallow	/shop/?mode
Disallow	/shop/index.php/
Disallow	/shop/*?SID=
Disallow	/shop/checkout/
Disallow	/shop/onestepcheckout/
Disallow	/shop/customer/
Disallow	/shop/customer/account/
Disallow	/shop/customer/account/login/
Disallow	/shop/catalogsearch/
Disallow	/shop/catalog/product_compare/
Disallow	/shop/catalog/category/view/
Disallow	/shop/catalog/product/view/
Disallow	/shop/cgi-bin/
Disallow	/shop/cleanup.php
Disallow	/shop/apc.php
Disallow	/shop/memcache.php
Disallow	/shop/phpinfo.php

Rule

Path

Disallow

/CVS

Disallow

/*.svn$

Disallow

/*.idea$

Disallow

/*.sql$

Disallow

/*.tgz$

Disallow

/shop/admin/

Disallow

/shop/app/

Disallow

/shop/downloader/

Disallow

/shop/errors/

Disallow

/shop/includes/

Disallow

/shop/lib/

Disallow

/shop/pkginfo/

Disallow

/shop/shell/

Disallow

/shop/var/

Disallow

/shop/api.php

Disallow

/shop/cron.php

Disallow

/shop/cron.sh

Disallow

/shop/error_log

Disallow

/shop/get.php

Disallow

/shop/install.php

Disallow

/shop/LICENSE.html

Disallow

/shop/LICENSE.txt

Disallow

/shop/LICENSE_AFL.txt

Disallow

/shop/README.txt

Disallow

/shop/RELEASE_NOTES.txt

Disallow

/shop/*?dir*

Disallow

/shop/*?dir=desc

Disallow

/shop/*?dir=asc

Disallow

/shop/*?limit=all

Disallow

/shop/*?mode*

Disallow

/shop/index.php/

Disallow

/shop/*?SID=

Disallow

/shop/checkout/

Disallow

/shop/onestepcheckout/

Disallow

/shop/customer/

Disallow

/shop/customer/account/

Disallow

/shop/customer/account/login/

Disallow

/shop/catalogsearch/

Disallow

/shop/catalog/product_compare/

Disallow

/shop/catalog/category/view/

Disallow

/shop/catalog/product/view/

Disallow

/shop/cgi-bin/

Disallow

/shop/cleanup.php

Disallow

/shop/apc.php

Disallow

/shop/memcache.php

Disallow

/shop/phpinfo.php

Other Records

Field	Value
crawl-delay	30

Field

Value

crawl-delay

msnbot-media

Rule	Path
Disallow	/

Rule

Path

Disallow

abonti

Rule	Path
Disallow	/

Rule

Path

Disallow

abonti/0.92

Rule	Path
Disallow	/

Rule

Path

Disallow

abot v1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

aboutthedomain

Rule	Path
Disallow	/

Rule

Path

Disallow

add catalog

Rule	Path
Disallow	/

Rule

Path

Disallow

add catalog/2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

advbot

Rule	Path
Disallow	/

Rule

Path

Disallow

advbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot/3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefs-bot/5.0

Rule	Path
Disallow	/

Rule

Path

Disallow

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aihitbot/2.9

Rule	Path
Disallow	/

Rule

Path

Disallow

anonymous/0.0

Rule	Path
Disallow	/

Rule

Path

Disallow

arachnida

Rule	Path
Disallow	/

Rule

Path

Disallow

associative spider

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

baidu spider

Rule	Path
Disallow	/

Rule

Path

Disallow

battleztar bazinga

Rule	Path
Disallow	/

Rule

Path

Disallow

battleztar bazinga/0.01

Rule	Path
Disallow	/

Rule

Path

Disallow

bdfetch

Rule	Path
Disallow	/

Rule

Path

Disallow

betabot

Rule	Path
Disallow	/

Rule

Path

Disallow

bieshu

Rule	Path
Disallow	/

Rule

Path

Disallow

bigli seo

Rule	Path
Disallow	/

Rule

Path

Disallow

blackboard safeassign

Rule	Path
Disallow	/

Rule

Path

Disallow

blazer 1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blp_bbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

boia-accessibility-agent/pr 1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bot for jce

Rule	Path
Disallow	/

Rule

Path

Disallow

bot/0.1 (bot for jce)

Rule	Path
Disallow	/

Rule

Path

Disallow

bpimagewalker

Rule	Path
Disallow	/

Rule

Path

Disallow

bpimagewalker/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

buibui-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

buibui-bot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ca-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

ca-crawler/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cakephp

Rule

Path

Disallow

calypso v/0.01

Rule

Path

Disallow

calypso

Rule

Path

Disallow

cb/nutch-1.7

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

ccbot/2.0

Rule

Path

Disallow

checkbot

Rule

Path

Disallow

checkgzipcompression.com

Rule

Path

Disallow

chushou

Rule

Path

Disallow

cloudservermarketspider

Rule

Path

Disallow

cloudservermarketspider/1.0

Rule

Path

Disallow

clushbot/3.x-binaryfury

Rule

Path

Disallow

cms crawler

Rule

Path

Disallow

cms crawler: http://www.cmscrawler.com

Rule

Path

Disallow

coccoc

Rule

Path

Disallow

coincornerbot

Rule

Path

Disallow

coincornerbot/1.1

Rule

Path

Disallow

copyscape

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

crazywebcrawler 0.9.0

Rule

Path

Disallow

crazywebcrawler 0.9.1

Rule

Path

Disallow

crazywebcrawler 0.9.7

Rule

Path

Disallow

crazywebcrawler

Rule

Path

Disallow

crazywebcrawler-spider

Rule

Path

Disallow

crowsnest

Rule

Path

Disallow

crowsnest/0.5

Rule

Path

Disallow

curious george - www.analyticsseo.com/crawler

Rule

Path

Disallow

curious george

Rule

Path

Disallow

cuwhois

Rule

Path

Disallow

cuwhois/1.0

Rule

Path

Disallow

dahoms

Rule

Path

Disallow

datagnionbot

Rule

Path

Disallow

deusu/5.0.2

Rule

Path

Disallow

digincore

Rule

Path

Disallow

digincore bot

Rule

Path

Disallow

dispatch/0.11.0

Rule

Path

Disallow

domain re-animator bot

Rule

Path

Disallow

domainappender /1.0

Rule

Path

Disallow

domainappender

Rule

Path

Disallow

domainsigmacrawler

Rule

Path

Disallow

domainsigmacrawler/0.1

Rule

Path

Disallow

domnutch

Rule

Path

Disallow

domnutch-bot

Rule

Path

Disallow

domnutch-bot/nutch

Rule

Path

Disallow

domnutch-bot/nutch-1.0

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

eccp/1.2.1

Rule

Path

Disallow

ecommercebot

Rule

Path

Disallow

enlle punto com/nutch-1.9

Rule

Path

Disallow

episerver link checker

Rule

Path

Disallow

euripbot

Rule

Path

Disallow

euripbot/2.0

Rule

Path

Disallow

evc/2.0

Rule

Path

Disallow

evc-batch

Rule

Path

Disallow

evc-batch/2.0

Rule

Path

Disallow

express webpictures

Rule

Path

Disallow

faraday v0.8.8

Rule

Path

Disallow

faraday

Rule

Path

Disallow

findxbot

Rule

Path

Disallow

findxbot/1.0

Rule

Path

Disallow

flamingo_searchengine

Rule

Path

Disallow

flipboard robot

Rule

Path

Disallow

getproxi.es-bot

Rule

Path

Disallow

getproxi.es-bot/1.1

Rule

Path

Disallow

gigablastopensource

Rule

Path

Disallow

gigablastopensource/1.0

Rule

Path

Disallow

girafabot

Rule

Path

Disallow

gluten free crawler

Rule

Path

Disallow

gluten free crawler/1.0

Rule

Path

Disallow

griffinbot

Rule

Path

Disallow

grifinbot/0.01

Rule

Path

Disallow

gwpimages

Rule

Path

Disallow

gwpimages/1.0

Rule

Path

Disallow

haiula

Rule

Path

Disallow

haiula/1.4

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

hivemind

Rule

Path

Disallow

hostharvest

Rule

Path

Disallow

hostharvest/0.4.28

Rule

Path

Disallow

hrcrawler

Rule

Path

Disallow

hrcrawler/2.0

Rule

Path

Disallow

http://git.io/tl_s2w

Rule

Path

Disallow

http://www.checkprivacy.or.kr:6600/rs/privacy_enfaq.jsp

Rule

Path

Disallow

hubspot links crawler 1.0

Rule

Path

Disallow

hubspot webcrawler

Rule

Path

Disallow

hubspot

Rule

Path

Disallow

hunchan

Rule

Path

Disallow

hypercrawl

Rule

Path

Disallow

hypercrawl/0.2

Rule

Path

Disallow

icap-iod

Rule

Path

Disallow

icc-crawler

Rule

Path

Disallow

icc-crawler/2.0

Rule

Path

Disallow

ichiro robot

Rule

Path

Disallow

image.coccoc/1.0

Rule

Path

Disallow

image2play

Rule

Path

Disallow

image2play/0.1

Rule

Path

Disallow

indy library

Rule

Path

Disallow

insightscollector

Rule

Path

Disallow

insightscollector/0.1

Rule

Path

Disallow

insightscollector/0.1beta

Rule

Path

Disallow

integrity/5

Rule

Path

Disallow

internaetboten

Rule

Path

Disallow

internaetboten/0.99

Rule

Path

Disallow

irl crawler

Rule

Path

Disallow

james bot - webcrawler

Rule

Path

Disallow

james bot

Rule

Path

Disallow

jamesbot

Rule

Path

Disallow

jetbrains 5.0

Rule

Path

Disallow

jetbrains

Rule

Path

Disallow

kraken

Rule

Path

Disallow

kraken/0.1

Rule

Path

Disallow

kyoto-tohoku-crawler/v1

Rule

Path

Disallow

larbin

Rule

Path

Disallow

lechenie

Rule

Path

Disallow

libwww-perl

Rule

Path

Disallow

link checker

Rule

Path

Disallow

link/1.0

Rule

Path

Disallow

linkcheck

Rule

Path

Disallow

linkcheckv3.0

Rule

Path

Disallow

linkdex

Rule

Path

Disallow

linkdex.com/v2.0

Rule

Path

Disallow

linkdex.com/v2.1

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

linkdexbot/2.0

Rule

Path

Disallow

linkdexbot/2.1

Rule

Path

Disallow

linkdexbot-mobile/2.1

Rule

Path

Disallow

linkpadbot

Rule

Path

Disallow

linkpadbot/1.06

Rule

Path

Disallow

linqiascrapebot

Rule

Path

Disallow

linqiascrapebot/1.0

Rule

Path

Disallow

lipperhey seo service

Rule

Path

Disallow

lipperhey

Rule

Path

Disallow

lipperhey-kaus-australis

Rule

Path

Disallow

lipperhey-kaus-australis/5.0

Rule

Path

Disallow

listicka

Rule

Path

Disallow

lssrocketcrawler

Rule

Path

Disallow

lssrocketcrawler/1.0 lightspeedsystems

Rule

Path

Disallow

lssrocketcrawler/1.0

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

lwnutch/nutch-1.4

Rule

Path

Disallow

mail.ru

Rule

Path

Disallow

mail.ru_bot

Rule

Path

Disallow

mail.ru_bot/2.0

Rule

Path

Disallow

mail.ru_bot/fast/2.0

Rule

Path

Disallow

md5sum

Rule

Path

Disallow

md5sum\x22

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

megaindex.ru/2.0

Rule

Path

Disallow

mezhpozvonochnoi

Rule

Path

Disallow

mike-crawler

Rule

Path

Disallow

mixbot

Rule

Path

Disallow

mixrankbot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

monkeybot/0.1

Rule

Path

Disallow

my crawler

Rule

Path

Disallow

my nutch spider/nutch-1.9

Rule

Path

Disallow

mycrowl/nutch-1.9

Rule

Path

Disallow

mygreatua/2.0

Rule

Path

Disallow

myiptest

Rule

Path

Disallow

nameprotect robot

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

netcraft spider

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

netlyzer fastprobe

Rule

Path

Disallow

netresearchserver

Rule

Path

Disallow

netresearchserver/4.0

Rule

Path

Disallow

nmap scripting engine

Rule

Path

Disallow

node.io

Rule

Path

Disallow

node.js

Rule

Path

Disallow

node/simplecrawler 0.5.2

Rule

Path

Disallow

node/simplecrawler

Rule

Path

Disallow

obot/2.3.1

Rule

Path

Disallow

omgilibot

Rule

Path

Disallow

omgilibot/0.4

Rule

Path

Disallow

online domain tools - online website link checker

Rule

Path

Disallow

online domain tools - online website link checker/1.2

Rule

Path

Disallow

openfind robot

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

openhosebot/2.1

Rule

Path

Disallow

openstat

Rule

Path

Disallow

openstat/0.1

Rule

Path

Disallow

optimizationcrawler

Rule

Path

Disallow

optimizationcrawler/0.2

Rule

Path

Disallow

page analyzer v4.0

Rule

Path

Disallow

page analyzer

Rule

Path

Disallow

pageanalyzer

Rule

Path

Disallow

pageanalyzer/1.1

Rule

Path

Disallow

pageanalyzer/1.5

Rule

Path

Disallow

pagesinventory

Rule

Path

Disallow

pagespeed/1.1 fetcher

Rule

Path

Disallow

pagespeed/1.1

Rule

Path

Disallow

pagespeedbot

Rule

Path

Disallow

perl lwp

Rule

Path

Disallow

phpcrawl

Rule

Path

Disallow

phpsitecheck 1.0

Rule

Path

Disallow

phpsitecheck

Rule

Path

Disallow

plukkie

Rule

Path

Disallow

pogs/2.0

Rule

Path

Disallow

powermarks

Rule

Path

Disallow

powerpivot

Rule

Path

Disallow

privacy_enfaq.jsp

Rule

Path

Disallow

prlog

Rule

Path

Disallow

prlog/1.0

Rule

Path

Disallow

publiclibraryarchive.org

Rule

Path

Disallow

publiclibraryarchive.org/1.0

Rule

Path

Disallow

pu_in crawler

Rule

Path

Disallow

putin

Rule

Path

Disallow

putin spider

Rule

Path

Disallow

qingdao

Rule

Path

Disallow

qlikview

Rule

Path

Disallow

quipu

Rule

Path

Disallow

quipu/1.0

Rule

Path

Disallow

quipu/2.0

Rule

Path

Disallow

r6_commentreader

Rule

Path

Disallow

r6_feedfetcher

Rule

Path

Disallow

riddler

Rule

Path

Disallow

rivalseek.com-bot

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

rogerbot/1.0

Rule

Path

Disallow

rootlink

Rule

Path

Disallow

ru_bot/2.0

Rule

Path

Disallow

scopia

Rule

Path

Disallow

scopia crawler

Rule

Path

Disallow

scopia crawler 1.0

Rule

Path

Disallow

scopia crawler 1.1

Rule

Path

Disallow

scopia crawler 1.2

Rule

Path

Disallow

scrapy

Rule

Path

Disallow

scrapy/0.16.5

Rule

Path

Disallow

scrapy/0.24.4

Rule

Path

Disallow

scrapy/0.24.5

Rule

Path

Disallow

scrapy/0.24.6

Rule

Path

Disallow

scrapy/1.0.1

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

screaming frog seo spider/2,55

Rule

Path

Disallow

screaming frog seo spider/2.55

Rule

Path

Disallow

screaming frog seo spider/3.1

Rule

Path

Disallow

screaming frog seo spider/3.3

Rule

Path

Disallow

screaming frog seo spider/4.1

Rule

Path

Disallow

screaming frog seo spider/5.0

Rule

Path

Disallow

screaming frog seo spider/5.1

Rule

Path

Disallow

screaming frog seo spider/5.1 beta 2

Rule

Path

Disallow

scrutiny/4

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

seodiver/1.0

Rule

Path

Disallow

seokicks

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

seolyticscrawler

Rule

Path

Disallow

seolyticscrawler/3.0

Rule

Path

Disallow

seoscanners

Rule

Path

Disallow

seoscanners.net/1

Rule

Path

Disallow

seostats 2.1.0

Rule

Path

Disallow

seosys/nutch-2.3

Rule

Path

Disallow

setcronjob/1.0

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

sheerboredom.experimental.robot

Rule

Path

Disallow

sheerboredom.experimental.robot/0.2

Rule

Path

Disallow

showyoubot

Rule

Path

Disallow

simplecrawler

Rule

Path

Disallow

sistrix crawler

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

sitebot

Rule

Path

Disallow

sitebot/0.1

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

siteexplorer/1.0

Rule

Path

Disallow

siteexplorer/1.0b

Rule

Path

Disallow

siteluxbot

Rule

Path

Disallow

siteluxbot/1.0

Rule

Path

Disallow

skimbot

Rule

Path

Disallow

skimbot/1.0

Rule

Path

Disallow

sky nutch crawler/nutch-1.9

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

smtbot/1.0

Rule

Path

Disallow

snk screenshot bot

Rule

Path

Disallow

snk screenshot bot/0.20

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

sogou web spider

Rule

Path

Disallow

spambayes

Rule

Path

Disallow

spambayes/1.1a3+

Rule

Path

Disallow

spbot

Rule

Path

Disallow

spbot/4.4.2

Rule

Path

Disallow

spiderbot

Rule

Path

Disallow

spiderling

Rule

Path

Disallow

spiderbot/nutch-1.7

Rule

Path

Disallow

spray-can

Rule

Path

Disallow

spray-can/1.2.1

Rule

Path

Disallow

ssg/3.0

Rule

Path

Disallow

statastico

Rule

Path

Disallow

statastico/4.0

Rule

Path

Disallow

steeler

Rule

Path

Disallow

steeler/3.5

Rule

Path

Disallow

stratagems kumo

Rule

Path

Disallow

stratagems

Rule

Path

Disallow

studiofaca search

Rule

Path

Disallow

studiofaca

Rule

Path

Disallow

sukibot

Rule

Path

Disallow

sukibot_heritrix

Rule

Path

Disallow

sukibot_heritrix/3.1.1

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

synapse

Rule

Path

Disallow

synthesio crawler release monalisa

Rule

Path

Disallow

tbot-nutch/nutch-1.10

Rule

Path

Disallow

traackr.com bot

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

trendiction-bot

Rule

Path

Disallow

truebot

Rule

Path

Disallow

truebot/1.0

Rule

Path

Disallow

tulipchain/5.xx

Rule

Path

Disallow

twmbot/0.1

Rule

Path

Disallow

typhoeus

Rule

Path

Disallow

ucmore crawler app

Rule

Path

Disallow

umbot-ln

Rule

Path

Disallow

umbot-ln/1.0

Rule

Path

Disallow

updown_tester

Rule

Path

Disallow

urlchecker

Rule

Path

Disallow

v1.0/1.2

Rule

Path

Disallow

w3af.org

Rule

Path

Disallow

wasalive

Rule

Path

Disallow

wasalive-bot

Rule

Path

Disallow

vbseo

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

wbsearchbot/1.1

Rule

Path

Disallow

wearenotevil

Rule

Path

Disallow

webalta

Rule

Path

Disallow

webalta crawler

Rule

Path

Disallow

web corpus crawler

Rule

Path

Disallow

webcookies

Rule

Path

Disallow

webcookies/1.0

Rule

Path

Disallow

webcopier vx.xa

Rule

Path

Disallow

webnest 0.9

Rule

Path

Disallow

webql

Rule

Path

Disallow

webscout

Rule

Path

Disallow

webscout/1.0

Rule

Path

Disallow

web-sniffer

Rule

Path

Disallow

web-sniffer/1.1.0

Rule

Path

Disallow

webster pro v3.4

Rule

Path

Disallow

webtarantula.com crawler

Rule

Path

Disallow

wecrawlforthepeace

Rule

Path

Disallow

vegebot

Rule

Path

Disallow

welikelinks

Rule

Path

Disallow

vericitecrawler

Rule

Path

Disallow

vericitecrawler/nutch-1.9

Rule

Path

Disallow

whatweb

Rule

Path

Disallow

whatweb/0.4.8-dev

Rule

Path

Disallow

visited by http://tools.geek-tools.org

Rule

Path

Disallow

voila robot

Rule

Path

Disallow

voltron

Rule

Path

Disallow

woobot

Rule

Path

Disallow

woobot/1.1

Rule

Path

Disallow

woobot/2.0

Rule

Path

Disallow

vorboss web crawler

Rule

Path

Disallow

vorboss web crawler/nutch-2.3

Rule

Path

Disallow

worldbrewbot

Rule

Path

Disallow

worldbrewbot/2.1

Rule

Path

Disallow

worldwebheritage.org

Rule

Path

Disallow

worldwebheritage.org/1.0

Rule

Path

Disallow

wscheck.com

Rule

Path

Disallow

wscheck.com/1.0.0

Rule

Path

Disallow

www.deadlinkchecker.com

Rule

Path

Disallow

www.petitsage.fr site detector 0.4

Rule

Path

Disallow

www-mechanize

Rule

Path

Disallow

www-mechanize/1.74

Rule

Path

Disallow

xenu link sleuth

Rule

Path

Disallow

xenu's link sleuth

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

xovibot/2.0

Rule

Path

Disallow

xspider

Rule

Path

Disallow

yandex robot

Rule

Path

Disallow

yandex

Rule

Path

Disallow

yandexbot/3.0

Rule

Path

Disallow

yandeximages/3.0

Rule

Path

Disallow

yetibot

Rule

Path

Disallow

yisouspider

Rule

Path

Disallow

yoozbot

Rule

Path

Disallow

yoozbot-2.2

Rule

Path

Disallow

zgrab/0.x

Rule

Path

Disallow

zzabmbot

Rule

Path

Disallow

zzabmbot/1.0

Rule

Path

Disallow

bingbot/2.0

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.simonetta.it/sitemap_index.xml

sitemap

https://www.simonetta.it/shop/sitemap/it/sitemap.xml

sitemap

https://www.simonetta.it/eu/shop/sitemap/eu/sitemap.xml

sitemap

https://www.simonetta.it/uk/shop/sitemap/uk/sitemap.xml

Comments

WORDPRESS
MAGENTO
Enable robots.txt rules for all crawlers
Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
Set a custom crawl rate if you're experiencing traffic problems with your server.
DEVELOPMENT RELATED SETTINGS
Do not crawl development files and folders: CVS, svn directories and dump files
GENERAL MAGENTO SETTINGS
Do not crawl Magento admin page
Do not crawl common Magento technical folders
Do not crawl common Magento files
MAGENTO SEO IMPROVEMENTS
Do not crawl sub category pages that are sorted or filtered.
Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
Do not crawl links with session IDs
Do not crawl checkout and user account pages
Do not crawl seach pages and not-SEO optimized catalog links
SERVER SETTINGS
Do not crawl common server technical folders and files
IMAGE CRAWLERS SETTINGS
Extra: Uncomment if you do not wish Google and Bing to index your images
User-agent: Googlebot-Image
Disallow: /

Warnings

14 invalid lines.

simonetta.itrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

Other Records

msnbot-media

abonti

abonti/0.92

abot v1.0

aboutthedomain

add catalog

add catalog/2.1

advbot

advbot/2.0

ahrefsbot

ahrefs-bot

ahrefsbot/1.0

ahrefs-bot/1.0

ahrefs-bot/2.0

ahrefs-bot/3.0

ahrefs-bot/4.0

ahrefs-bot/5.0

aihitbot

aihitbot/2.9

anonymous/0.0

arachnida

associative spider

baiduspider

baidu spider

battleztar bazinga

battleztar bazinga/0.01

bdfetch

betabot

bieshu

bigli seo

blackboard safeassign

blazer 1.0

blexbot

blexbot/1.0

blp_bbot

blp_bbot/0.1

boia-accessibility-agent/pr 1.0

bot for jce

bot/0.1 (bot for jce)

bpimagewalker

bpimagewalker/2.0

bubing

buibui-bot

buibui-bot/1.0

ca-crawler

ca-crawler/1.0

cakephp

calypso v/0.01

calypso

cb/nutch-1.7

ccbot

ccbot/2.0

checkbot

checkgzipcompression.com

chushou

cloudservermarketspider

cloudservermarketspider/1.0

clushbot/3.x-binaryfury

cms crawler

cms crawler: http://www.cmscrawler.com

coccoc

coincornerbot

coincornerbot/1.1

copyscape

crawler4j

crazywebcrawler 0.9.0

crazywebcrawler 0.9.1

crazywebcrawler 0.9.7

crazywebcrawler

crazywebcrawler-spider

crowsnest

crowsnest/0.5

curious george - www.analyticsseo.com/crawler

simonetta.it
robots.txt