materialesalicante.com
robots.txt

Robots Exclusion Standard data for materialesalicante.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	materialesalicante.com
Base Domain	materialesalicante.com
Scan Status	Ok
Last Scan	2024-11-15T12:44:59+00:00
Next Scan	2024-11-22T12:44:59+00:00

Last Scan

Scanned	2024-11-15T12:44:59+00:00
URL	https://materialesalicante.com/robots.txt
Domain IPs	2a02:4780:84:cd6e:b008:36fc:f548:1de8, 84.32.84.141
Response IP	77.37.66.43
Found	Yes
Hash	7a73fc31d0019c9a9072550bc90c26bdbe8f10daa1c1d2e87692cc7c785573af
SimHash	6354fcc089c4

Groups

*

Rule	Path
Disallow	/wp-admin/
Allow	/wp-admin/admin-ajax.php
Disallow	/page/*
Disallow	//page/
Disallow	/pide-prespuesto-online/
Disallow	*/feed/
Disallow	?feed
Disallow	/feed/
Disallow	/feed/*
Disallow	/author/*
Disallow	/2014/*
Disallow	/2015/*
Disallow	/2016/*
Disallow	/2017/*
Disallow	/2018/*
Disallow	/2019/*
Disallow	/2020/*
Disallow	/2021/*
Disallow	/2022/*
Disallow	/2023/*
Disallow	/?s=
Disallow	/search
Disallow	/?page_id=8
Disallow	/carro
Disallow	/tienda/*
Disallow	/precios/*
Disallow	/tag/*
Disallow	/?p=*
Disallow	/comment*
Disallow	/*?nonamp=1
Disallow	/wp-content/plugins/link-juice-optimizer/public/js/
Disallow	/?
Disallow	/trackback
Disallow	/*trackback
Disallow	/trackback
Disallow	/*/trackback
Allow	/feed/$
Disallow	/feed/
Disallow	/comments/feed/
Disallow	/*/feed/$
Disallow	/*/feed/rss/$
Disallow	/*/trackback/$
Disallow	///feed/$
Disallow	///feed/rss/$
Disallow	///trackback/$
Disallow	///*/feed/$
Disallow	///*/feed/rss/$
Disallow	///*/trackback/$

Rule

Path

Disallow

/wp-admin/

Allow

/wp-admin/admin-ajax.php

Disallow

/page/*

Disallow

/*/page/*

Disallow

/pide-prespuesto-online/

Disallow

*/feed/

Disallow

*?feed*

Disallow

/feed/

Disallow

/feed/*

Disallow

/author/*

Disallow

/2014/*

Disallow

/2015/*

Disallow

/2016/*

Disallow

/2017/*

Disallow

/2018/*

Disallow

/2019/*

Disallow

/2020/*

Disallow

/2021/*

Disallow

/2022/*

Disallow

/2023/*

Disallow

/?s=

Disallow

/search

Disallow

/?page_id=8

Disallow

/carro

Disallow

/tienda/*

Disallow

/precios/*

Disallow

/tag/*

Disallow

/?p=*

Disallow

/comment*

Disallow

/*?nonamp=1

Disallow

/wp-content/plugins/link-juice-optimizer/public/js/

Disallow

/*?*

Disallow

/trackback

Disallow

/*trackback

Disallow

/*trackback*

Disallow

/*/trackback

Allow

/feed/$

Disallow

/feed/

Disallow

/comments/feed/

Disallow

/*/feed/$

Disallow

/*/feed/rss/$

Disallow

/*/trackback/$

Disallow

/*/*/feed/$

Disallow

/*/*/feed/rss/$

Disallow

/*/*/trackback/$

Disallow

/*/*/*/feed/$

Disallow

/*/*/*/feed/rss/$

Disallow

/*/*/*/trackback/$

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	20

Field

Value

crawl-delay

msnbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	20

Field

Value

crawl-delay

slurp

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	20

Field

Value

crawl-delay

addthis.com

Rule	Path
Disallow	/

Rule

Path

Disallow

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

acoonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

bdcbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bender

Rule	Path
Disallow	/

Rule

Path

Disallow

bixocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

blackwidow

Rule	Path
Disallow	/

Rule

Path

Disallow

bl.uk_lddc_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

chinaclaw

Rule	Path
Disallow	/

Rule

Path

Disallow

cliqzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

cncdialer

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler4j

Rule	Path
Disallow	/

Rule

Path

Disallow

crystalsemanticsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

custo

Rule	Path
Disallow	/

Rule

Path

Disallow

cyberalert

Rule	Path
Disallow	/

Rule

Path

Disallow

digext

Rule	Path
Disallow	/

Rule

Path

Disallow

discobot

Rule	Path
Disallow	/

Rule

Path

Disallow

discoverybot

Rule	Path
Disallow	/

Rule

Path

Disallow

dloader

Rule	Path
Disallow	/

Rule

Path

Disallow

dloader(naverrobot)

Rule	Path
Disallow	/

Rule

Path

Disallow

doc

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

download ninja

Rule	Path
Disallow	/

Rule

Path

Disallow

dts agent

Rule	Path
Disallow	/

Rule

Path

Disallow

disco

Rule	Path
Disallow	/

Rule

Path

Disallow

emailsiphon

Rule	Path
Disallow	/

Rule

Path

Disallow

emailwolf

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms

Rule	Path
Disallow	/

Rule

Path

Disallow

fairshare

Rule	Path
Disallow	/

Rule

Path

Disallow

fasterfox

Rule	Path
Disallow	/

Rule

Path

Disallow

fetch

Rule	Path
Disallow	/

Rule

Path

Disallow

flamingo_searchengine

Rule	Path
Disallow	/

Rule

Path

Disallow

gaisbot

Rule	Path
Disallow	/

Rule

Path

Disallow

genieo

Rule	Path
Disallow	/

Rule

Path

Disallow

gigabot

Rule	Path
Disallow	/

Rule

Path

Disallow

grub-client

Rule	Path
Disallow	/

Rule

Path

Disallow

harvest

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix/3.3.0

Rule

Path

Disallow

heritrix

Rule

Path

Disallow

hloader

Rule

Path

Disallow

httrack

Rule

Path

Disallow

httplib

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

integromedb

Rule

Path

Disallow

istellabot

Rule

Path

Disallow

jikespider

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

jyxobot

Rule

Path

Disallow

kenjin spider

Rule

Path

Disallow

keyword density/0.9

Rule

Path

Disallow

k2spider

Rule

Path

Disallow

kimengi

Rule

Path

Disallow

kimengi/nineconnections.com

Rule

Path

Disallow

larbin

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

lexxebot/1.0

Rule

Path

Disallow

libwww

Rule

Path

Disallow

linko

Rule

Path

Disallow

livelapbot

Rule

Path

Disallow

magpie-crawler

Rule

Path

Disallow

maxthon

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

mister pix

Rule

Path

Disallow

metauri

Rule

Path

Disallow

microsoft.url.control

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

moget

Rule

Path

Disallow

moreover

Rule

Path

Disallow

moreoverbot

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

nabot

Rule

Path

Disallow

naverbot

Rule

Path

Disallow

nerdbynature.bot

Rule

Path

Disallow

netestate ne crawler

Rule

Path

Disallow

netseer crawler

Rule

Path

Disallow

newscan

Rule

Path

Disallow

nextgensearchbot

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

npbot

Rule

Path

Disallow

nutch

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

offline navigator

Rule

Path

Disallow

omgilibot

Rule

Path

Disallow

openfind

Rule

Path

Disallow

orthogaffe

Rule

Path

Disallow

piplbot

Rule

Path

Disallow

pixray-seeker

Rule

Path

Disallow

propowerbot/2.14

Rule

Path

Disallow

proximic

Rule

Path

Disallow

psbot

Rule

Path

Disallow

queryseekerspider

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

rma

Rule

Path

Disallow

seokicks

Rule

Path

Disallow

seokicks-robot

Rule

Path

Disallow

sitebot

Rule

Path

Disallow

sitebot/0.1

Rule

Path

Disallow

sitecheck.internetseer.com

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

slurp

Rule

Path

Disallow

sproose/0.1

Rule

Path

Disallow

so-net

Rule

Path

Disallow

sogou

Rule

Path

Disallow

spbot

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

steeler

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

szukacz

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

ucweb

Rule

Path

Disallow

unisterbot

Rule

Path

Disallow

volcano

Rule

Path

Disallow

webcapture

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webdown

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wget

Rule

Path

Disallow

wiseguys robot

Rule

Path

Disallow

wofindeich robot 1.0

Rule

Path

Disallow

woobot

Rule

Path

Disallow

woobot/2.1

Rule

Path

Disallow

xenu

Rule

Path

Disallow

yandexbot

Rule

Path

Disallow

yandeximages

Rule

Path

Disallow

zeus

Rule

Path

Disallow

zmeu

Rule

Path

Disallow

googlebot

Rule

Path

Allow

/*.css$

Allow

/*.js$

Allow

/*.jpg$

Allow

/*.jpeg$

Allow

/*.png$

Allow

/*.gif$

Allow

/*.webp$

Other Records

Field

Value

sitemap

https://materialesalicante.com/sitemap.xml

Comments

Bloqueo de las URL dinamicas
Bloqueo de trackbacks
Bloqueo de feeds para crawlers
RALENTIZAMOS BOTS QUE DAN PROBLEMAS
BOTS POCO UTILES

Warnings

2 invalid lines.

materialesalicante.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

noxtrumbot

Other Records

msnbot

Other Records

slurp

Other Records

addthis.com

admantx

ahrefsbot

acoonbot

alexibot

baiduspider

bdcbot

bender

bixocrawler

blackwidow

bl.uk_lddc_bot

blexbot

bubing

chinaclaw

cliqzbot

cncdialer

crawler4j

crystalsemanticsbot

custo

cyberalert

digext

discobot

discoverybot

dloader

dloader(naverrobot)

doc

dotbot

download ninja

dts agent

disco

emailsiphon

emailwolf

exabot

extractorpro

ezooms

fairshare

fasterfox

fetch

flamingo_searchengine

gaisbot

genieo

gigabot

grub-client

harvest

heritrix/3.3.0

heritrix

hloader

httrack

httplib

humanlinks

ia_archiver

infonavirobot

integromedb

istellabot

jikespider

jennybot

jyxobot

kenjin spider

keyword density/0.9

k2spider

kimengi

kimengi/nineconnections.com

larbin

lexibot

lexxebot/1.0

libwww

linko

livelapbot

materialesalicante.com
robots.txt