pegperego.com
robots.txt
Robots Exclusion Standard data for pegperego.com
Resource Scan
Scan Details
| Site Domain | pegperego.com |
| Base Domain | pegperego.com |
| Scan Status | Ok |
| Last Scan | 2024-09-26T12:42:44+00:00 |
| Next Scan | 2024-10-26T12:42:44+00:00 |
Last Scan
| Scanned | 2024-09-26T12:42:44+00:00 |
| URL | https://www.pegperego.com/robots.txt |
| Domain IPs | 151.101.1.124 |
| Response IP | 151.101.1.124 |
| Found | Yes |
| Hash | f0528953711edf208d7762600d0abcdd9859db1672b39b5ea54f27d19ef0d914 |
| SimHash | 477bd25ba9b2 |
Groups
*
| Rule | Path |
|---|---|
| Allow | |
| Disallow | /index.php/ |
| Disallow | /app/ |
| Disallow | /lib/ |
| Disallow | /*.php$ |
| Disallow | /pkginfo/ |
| Disallow | /report/ |
| Disallow | /var/ |
| Disallow | /catalog/ |
| Disallow | /customer/ |
| Disallow | /sendfriend/ |
| Disallow | /review/ |
| Disallow | /*SID%3D |
| Disallow | */catalogsearch/* |
| Disallow | */checkout/* |
| Disallow | */customer/account/* |
| Disallow | *car_seat%3D* |
| Disallow | *color%3D* |
| Disallow | *bassinet%3D* |
| Disallow | *price%3D* |
| Disallow | *cat%3D* |
| Disallow | *product_list_order%3D* |
| Disallow | *stroller%3D* |
| Disallow | */product_compare/* |
a1 sitemap generator
abachobot
abcdatos botlink
aboundexbot
aboutusbot
accoona-ai-agent
addsugarspiderbot
adidxbot
ahoy! the homepage finder
aitcsrobot/1.1
alexibot
amznkassocbot
aqua_products
arachnophilia
architextspider
aspider/0.09
asterias
auresys/1.0
awariorssbot
awariosmartbot
b2w/0.1
backdoorbot/1.0
backrub/.
becomebot
beslistbot
big brother
bizbot003
black hole
blackwidow
blexbot
blowfish
blowfish/1.0
blp_bbot
bookmark search tool
botalot
botrighthere
builtbottough
bullseye/1.0
bunnyslippers
businessdbbot
cactvs chemistry spider
catchbot
ccbot
cegbfeieh
checkbot/x.xx lwp/5.x
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
coccoc
combine/0.0
conceptbot/0.3
converacrawler
copernic
copyrightcheck
cosmos
covarioids
crescent
crescent internet toolpak http ole control v.1.0
custo
cyberpatrol sitecat webbot
cyberspyder/2.1
daumoa
deweb/1.01
disco
disco pump 3.0
disco pump 3.2
discobot
discofinder
dittospyder
dotbot
download demon
download demon/3.2.0.8
download demon/3.5.0.11
download ninja
dumbot
ecatch
ecatch/3.0
eirgrabber
email exractor
emailcollector
emailsiphon
emailwolf
enigmabot
envolk
erocrawler
exabot
express webpictures
extractorpro
eyenetie
ezooms
fairad client
fdm 3.x
fdse robot
felixide/1.0
fido/0.9 harvest/1.4.pl2
fish-search-robot
flaming attackbot
flashget
flashget webwasher 3.2
flaxcrawler
foobot
freecrawl
frontpage
gaisbot
genieo
getright/4.2
getweb!
gigabot
girafabot
golem/1.1
grabber
grabnet
grafula
grapeshot
gromit/1.0
grub
grub-client
gsa-crawler
gslfbot
happyfunbot
harvest/1.5
hatena antenna
hazel's ferret web hopper
heritrix
hloader
hmview
httplib
huaweisymantecspider
humanlinks
ia_archiver
image stripper
image sucker
inagist.com url crawler
incywincy/1.0b1
indy library
infonavirobot
informant
ingrid/0.1
intelium_bot
interget
internet ninja 6.0
irlbot
iron33/1.0.2
israelisearch/1.0
istellabot
iti spider
jennybot
jetcar
joc web spider
jubiirobot
jumpstation
katipo/1.0
kenjin spider
keyword density/0.9
labelgrab/1.1
larbin
leechftp
lemurwebcrawler
lexibot
libweb/clshttp
libwww-perl
linguee bot
linkdexbot/2.1
linkedinbot
linkextractorpro
linklooker
linkscan/8.1a unix
linkwalker
lnspiderguy
looksmart
lwp-trivial
lwp-trivial/1.34
magpie-crawler
mass downloader/2.2
mata hari
mediafox/x.y
merzscope
metagopher
metamojicrawler
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
midown tool
miixpc
miixpc/4.2
mister pix
mnogosearch
moget
moget/2.1
momspider/1.00 libwww-perl/0.40
motor/0.2
msiecrawler
msrbot
msr-isrccrawler
navroad
nearsite
neosciocrawler
net vampire/3.0
netants
netcarta cyberpilot pro
netmechanic
netscoop/1.0 libwww/5.0a
netspider
netzip
nhsewalker/3.0
nicerspro
nomad-v2.x
northstar
npbot
nutch
obot
occam/1.0
octopus
offline explorer
ogspider
omniexplorer_bot
open text site crawler v1.0
openacoon
openbot
openfind
openfind data gathere
oracle ultra search
pagegrabber
pagepeeker
papa foto
pavuk
pcbrowser
perman
pgp-ka/1.2
plukkie
propowerbot/2.14
prowebwalker
proximic
psbot
python-urllib
queryn metasearch
r6_commentreader
r6_feedfetcher
radiation retriever 1.1
realdownload/4.0.0.42
reget
repomonkey
repomonkey bait & tackle/v1.01
resume robot
rma
roverbot
ruby
safetynet robot 0.1
scoutjet
searchpreview
senrigan/xxxxxx
seznambot
sitesnagger
slysearch
smartdownload
snooper/b97_01
sogou web spider
solbot/1.0 lwp/5.07
sootle
sosospider
spankbot
spanner
spanner/1.0 (linux 2.0.27 i586)
spbot
speedy spider
spyder3.microsys.com
superbot
superhttp/1.0
surfbot
surveybot
suzuran
szukacz/1.4
takeout
teleport
teleportpro
telesoft
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
turnitinbot
twiceler
ucsd-crawler
unisterbot
unwindfetchor/1.0
url control
url_spider_pro
urlck/1.2.3
urly warning
valkyrie/1.0 libwww-perl/0.40
vbseo
vci
vci webviewer vci webviewer win32
voideye
voilabot
wbsearchbot
web image collector
web sucker
webauto
webbandit
webbandit/3.50
webcopier
webenhancer
webferret
webfetch
webleacher
weblexbot
webmasterworld extractor
webmasterworldforumbot
webreaper
websauger
website quester
webster pro
webstripper
webvac
webwalk
webwatch
webwhacker
webzip
webzip/4.0
wget
wget/1.5.3
wget/1.6
whowhere robot
widow
www-collector-e
xaldon webspider
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
zyborg
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 5 invalid lines.
Comments