lapedrera.com
robots.txt

Robots Exclusion Standard data for lapedrera.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	lapedrera.com
Base Domain	lapedrera.com
Scan Status	Ok
Last Scan	2024-06-01T03:30:06+00:00
Next Scan	2024-07-01T03:30:06+00:00

Last Scan

Scanned	2024-06-01T03:30:06+00:00
URL	https://lapedrera.com/robots.txt
Redirect	https://www.lapedrera.com/robots.txt
Redirect Domain	www.lapedrera.com
Redirect Base	lapedrera.com
Domain IPs	213.192.239.75
Redirect IPs	213.192.239.75
Response IP	213.192.239.75
Found	Yes
Hash	69e50909a7b1e3ee36f618aab7efe9b020890f387c0e3c9b5b8952bf9cec156d
SimHash	b1d6b61bc820

Groups

*

Rule	Path
Allow	/core/*.css$
Allow	/core/*.css?
Allow	/core/*.js$
Allow	/core/*.js?
Allow	/core/*.gif
Allow	/core/*.jpg
Allow	/core/*.jpeg
Allow	/core/*.png
Allow	/core/*.svg
Allow	/profiles/*.css$
Allow	/profiles/*.css?
Allow	/profiles/*.js$
Allow	/profiles/*.js?
Allow	/profiles/*.gif
Allow	/profiles/*.jpg
Allow	/profiles/*.jpeg
Allow	/profiles/*.png
Allow	/profiles/*.svg
Disallow	/core/
Disallow	/profiles/
Disallow	/README.txt
Disallow	/web.config
Disallow	/admin/
Disallow	/comment/reply/
Disallow	/filter/tips
Disallow	/node/add/
Disallow	/search/
Disallow	/user/register/
Disallow	/user/password/
Disallow	/user/login/
Disallow	/user/logout/
Disallow	/index.php/admin/
Disallow	/index.php/comment/reply/
Disallow	/index.php/filter/tips
Disallow	/index.php/node/add/
Disallow	/index.php/search/
Disallow	/index.php/user/password/
Disallow	/index.php/user/register/
Disallow	/index.php/user/login/
Disallow	/index.php/user/logout/

Rule

Path

Allow

/core/*.css$

Allow

/core/*.css?

Allow

/core/*.js$

Allow

/core/*.js?

Allow

/core/*.gif

Allow

/core/*.jpg

Allow

/core/*.jpeg

Allow

/core/*.png

Allow

/core/*.svg

Allow

/profiles/*.css$

Allow

/profiles/*.css?

Allow

/profiles/*.js$

Allow

/profiles/*.js?

Allow

/profiles/*.gif

Allow

/profiles/*.jpg

Allow

/profiles/*.jpeg

Allow

/profiles/*.png

Allow

/profiles/*.svg

Disallow

/core/

Disallow

/profiles/

Disallow

/README.txt

Disallow

/web.config

Disallow

/admin/

Disallow

/comment/reply/

Disallow

/filter/tips

Disallow

/node/add/

Disallow

/search/

Disallow

/user/register/

Disallow

/user/password/

Disallow

/user/login/

Disallow

/user/logout/

Disallow

/index.php/admin/

Disallow

/index.php/comment/reply/

Disallow

/index.php/filter/tips

Disallow

/index.php/node/add/

Disallow

/index.php/search/

Disallow

/index.php/user/password/

Disallow

/index.php/user/register/

Disallow

/index.php/user/login/

Disallow

/index.php/user/logout/

a1 sitemap generator

Rule	Path
Disallow	/

Rule

Path

Disallow

addsugarspiderbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

arachnophilia

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

backrub/.

Rule	Path
Disallow	/

Rule

Path

Disallow

big brother

Rule	Path
Disallow	/

Rule

Path

Disallow

black hole

Rule	Path
Disallow	/

Rule

Path

Disallow

black.hole

Rule	Path
Disallow	/

Rule

Path

Disallow

blackwidow

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish

Rule	Path
Disallow	/

Rule

Path

Disallow

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

cactvs chemistry spider

Rule	Path
Disallow	/

Rule

Path

Disallow

catchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cegbfeieh

Rule	Path
Disallow	/

Rule

Path

Disallow

changedetection

Rule	Path
Disallow	/

Rule

Path

Disallow

checkbot/x.xx lwp/5.x

Rule	Path
Disallow	/

Rule

Path

Disallow

cheesebot

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypicker

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerse/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

chinaclaw

Rule	Path
Disallow	/

Rule

Path

Disallow

compspybot

Rule	Path
Disallow	/

Rule

Path

Disallow

conceptbot/0.3

Rule	Path
Disallow	/

Rule

Path

Disallow

copernic

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

cosmos

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent internet toolpak http ole control v.1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

curious george

Rule	Path
Disallow	/

Rule

Path

Disallow

cyberpatrol sitecat webbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cyberspyder/2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

cybeye.com

Rule	Path
Disallow	/

Rule

Path

Disallow

doc

Rule	Path
Disallow	/

Rule

Path

Disallow

discofinder

Rule	Path
Disallow	/

Rule

Path

Disallow

dittospyder

Rule	Path
Disallow	/

Rule

Path

Disallow

docomo

Rule	Path
Disallow	/

Rule

Path

Disallow

download demon

Rule	Path
Disallow	/

Rule

Path

Disallow

download ninja

Rule	Path
Disallow	/

Rule

Path

Disallow

dumbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ecatch

Rule	Path
Disallow	/

Rule

Path

Disallow

eirgrabber

Rule

Path

Disallow

emailcollector

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

enigmabot

Rule

Path

Disallow

exb language crawler

Rule

Path

Disallow

explorersearch

Rule

Path

Disallow

express webpictures

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

fish-search-robot

Rule

Path

Disallow

flaming attackbot

Rule

Path

Disallow

flamingo_searchengine

Rule

Path

Disallow

flashget

Rule

Path

Disallow

foobot

Rule

Path

Disallow

freecrawl

Rule

Path

Disallow

fetch

Rule

Path

Disallow

gaisbot

Rule

Path

Disallow

gcreep/1.0

Rule

Path

Disallow

genieo

Rule

Path

Disallow

genio

Rule

Path

Disallow

getright

Rule

Path

Disallow

getweb!

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

golem/1.1

Rule

Path

Disallow

grabnet

Rule

Path

Disallow

grub-client

Rule

Path

Disallow

gsa-crawler

Rule

Path

Disallow

happyfunbot

Rule

Path

Disallow

httrack

Rule

Path

Disallow

httrack 3.0

Rule

Path

Disallow

huaweisymantecspider

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

image stripper

Rule

Path

Disallow

image sucker

Rule

Path

Disallow

inagist.com url crawler

Rule

Path

Disallow

incywincy/1.0b1

Rule

Path

Disallow

indy library

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

informant

Rule

Path

Disallow

interget

Rule

Path

Disallow

internet ninja

Rule

Path

Disallow

israelisearch/1.0

Rule

Path

Disallow

iti spider

Rule

Path

Disallow

jikespider

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

joc web spider

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

jumpstation

Rule

Path

Disallow

jubiirobot

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword density/0.9

Rule

Path

Disallow

keyword.density

Rule

Path

Disallow

labelgrab/1.1

Rule

Path

Disallow

larbin

Rule

Path

Disallow

leechftp

Rule

Path

Disallow

lexxebot

Rule

Path

Disallow

libwww

Rule

Path

Disallow

libcrawl

Rule

Path

Disallow

linkdex

Rule

Path

Disallow

linklooker

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linko

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwnutch

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

mass downloader

Rule

Path

Disallow

mata hari

Rule

Path

Disallow

mata.hari

Rule

Path

Disallow

merzscope

Rule

Path

Disallow

metagopher

Rule

Path

Disallow

midown tool

Rule

Path

Disallow

moget

Rule

Path

Disallow

momspider/1.00 libwww-perl/0.40

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

navroad

Rule

Path

Disallow

net vampire

Rule

Path

Disallow

neosciocrawler

Rule

Path

Disallow

netcarta cyberpilot pro

Rule

Path

Disallow

npbot

Rule

Path

Disallow

netzip

Rule

Path

Disallow

netzip downloader 1.0 win32(nov 12 1998)

Rule

Path

Disallow

occam/1.0

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

open text site crawler v1.0

Rule

Path

Disallow

openbot

Rule

Path

Disallow

openwebindex

Rule

Path

Disallow

orthogaffe

Rule

Path

Disallow

pagegrabber

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

papa foto

Rule

Path

Disallow

pagepeeker

Rule

Path

Disallow

propowerbot/2.14

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

psbot

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

queryn metasearch

Rule

Path

Disallow

radiation retriever 1.1

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

reget

Rule

Path

Disallow

rediffnewsbot

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

resume robot

Rule

Path

Disallow

rma

Rule

Path

Disallow

roverbot

Rule

Path

Disallow

safetynet robot 0.1

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

senrigan/xxxxxx

Rule

Path

Disallow

seoengworldbot

Rule

Path

Disallow

seolyticscrawler/3.0

Rule

Path

Disallow

seoscanners

Rule

Path

Disallow

seoscanners.net

Rule

Path

Disallow

seoscanners.net/1

Rule

Path

Disallow

shopwiki

Rule

Path

Disallow

showyoubot

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

snooper/b97_01

Rule

Path

Disallow

solbot/1.0 lwp/5.07

Rule

Path

Disallow

sosospider

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

spanner

Rule

Path

Disallow

sogou

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

spyder3.microsys.com

Rule

Path

Disallow

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

takeout

Rule

Path

Disallow

teleport pro

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

the intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

turingos

Rule

Path

Disallow

ubicrawler

Rule

Path

Disallow

url control

Rule

Path

Disallow

urlspiderpro

Rule

Path

Disallow

urly warning

Rule

Path

Disallow

vbseo

Rule

Path

Disallow

voideye

Rule

Path

Disallow

web image collector

Rule

Path

Disallow

web sucker

Rule

Path

Disallow

web.image.collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webcapture 2.0

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webcopy/

Rule

Path

Disallow

webcrawler/3.0 robot libwww/5.0a

Rule

Path

Disallow

webdatacentrebot

Rule

Path

Disallow

webemailextrac.*

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webferret

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webgo is

Rule

Path

Disallow

weblayers/0.0

Rule

Path

Disallow

webleacher

Rule

Path

Disallow

weblinker/0.0 libwww-perl/0.1

Rule

Path

Disallow

webmasterworld extractor

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

website extractor

Rule

Path

Disallow

website quester

Rule

Path

Disallow

webster pro

Rule

Path

Disallow

webster.pro

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webwatch

Rule

Path

Disallow

webwhacker

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wget

Rule

Path

Disallow

widow

Rule

Path

Disallow

www collector

Rule

Path

Disallow

www.freeloader.com.

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

xaldon webspider

Rule

Path

Disallow

xenu's

Rule

Path

Disallow

xenu's link sleuth 1.1c

Rule

Path

Disallow

xget/0.7

Rule

Path

Disallow

youdaobot

Rule

Path

Disallow

yolinkbot

Rule

Path

Disallow

zealbot

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zeus link scout

Rule

Path

Disallow

zao

Rule

Path

Disallow

zyborg

Rule

Path

Disallow

sitecheck.internetseer.com

Rule

Path

Disallow

sindice-site-manager

Rule

Path

Disallow

repparser

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

scanmine

Rule

Path

Disallow

unisterbot

Rule

Path

Disallow

wocbot

Rule

Path

Disallow

woriobot

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

zibber-v0.1(www.zibb.com/crawler/)

Rule

Path

Disallow

Comments

robots.txt
This file is to prevent the crawling and indexing of certain parts
of your site by web crawlers and spiders run by sites like Yahoo!
and Google. By telling these "robots" where not to go on your site,
you save bandwidth and server resources.
This file will be ignored unless it is at the root of your host:
Used: http://example.com/robots.txt
Ignored: http://example.com/site/robots.txt
For more information about the robots.txt standard, see:
http://www.robotstxt.org/robotstxt.html
CSS, JS, Images
Directories
Files
Paths (clean URLs)
Paths (no clean URLs)

Warnings

4 invalid lines.

lapedrera.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

a1 sitemap generator

addsugarspiderbot

alexibot

ahrefsbot

arachnophilia

backdoorbot

backdoorbot/1.0

backrub/.

big brother

black hole

black.hole

blackwidow

blexbot

blowfish

botalot

bullseye

bullseye/1.0

bunnyslippers

cactvs chemistry spider

catchbot

cegbfeieh

changedetection

checkbot/x.xx lwp/5.x

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

chinaclaw

compspybot

conceptbot/0.3

copernic

copyrightcheck

cosmos

crescent

crescent internet toolpak http ole control v.1.0

curious george

cyberpatrol sitecat webbot

cyberspyder/2.1

cybeye.com

doc

discofinder

dittospyder

docomo

download demon

download ninja

dumbot

ecatch

eirgrabber

emailcollector

emailsiphon

emailwolf

enigmabot

exb language crawler

explorersearch

express webpictures

extractorpro

fish-search-robot

flaming attackbot

flamingo_searchengine

flashget

foobot

freecrawl

fetch

gaisbot

gcreep/1.0

genieo

genio

getright

getweb!

go-ahead-got-it

golem/1.1

grabnet

grub-client

gsa-crawler

lapedrera.com
robots.txt