Log In Sign Up

clubvwgolf.com
robots.txt

Robots Exclusion Standard data for clubvwgolf.com

Resource Scan

Scan Details

Site Domain clubvwgolf.com
Base Domain clubvwgolf.com
Scan Status Ok
Last Scan2024-11-11T11:36:14+00:00
Next Scan 2024-11-18T11:36:14+00:00

Last Scan

Scanned2024-11-11T11:36:14+00:00
URL https://clubvwgolf.com/robots.txt
Redirect https://portal.clubvwgolf.com/robots.txt
Redirect Domain portal.clubvwgolf.com
Redirect Base clubvwgolf.com
Domain IPs 2001:41d0:301:4::23, 46.105.204.23
Redirect IPs 2001:41d0:301:4::23, 46.105.204.23
Response IP 46.105.204.23
Found Yes
Hash 8349504f52604605c8456bf9d14da60005a1f140d6fdd6f537f676e0b49767e8
SimHash 63d7fa52ca37

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /feed/
Disallow */feed/

*

Rule Path
Disallow /trackback
Disallow /*trackback
Disallow /*trackback*
Disallow /*/trackback
Disallow /*/trackback/$
Disallow /*/*/trackback/$

*

Rule Path
Allow /feed/$
Disallow /feed/
Disallow */feed/
Disallow */feed/*
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow */page/*
Disallow /*/page/
Disallow *index.rss
Disallow /comments/feed/
Disallow /comments/
Disallow /seo/page/*
Disallow /xmlrpc.php
Disallow /?attachment_id*
Disallow /*/uploads/*
Disallow */tag/*
Disallow */author/*
Disallow /foro/members/*
Disallow /foro/search/*
Disallow /foro/data/attachments/*
Disallow /foro/posts/*
Disallow */reply*
Disallow /politica-de-proteccion-de-datos/
Disallow /aviso-legal/
Disallow /readme.html
Disallow /privacidad
Allow /*.js$

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

msnbot-media

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

adidxbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bingpreview

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Allow /

yandexbot

Rule Path
Allow /

yandeximages

Rule Path
Allow /

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

ahrefsbot
archive.org_bot
semrushbot

Rule Path
Disallow /

a1 sitemap generator

Rule Path
Disallow /

addsugarspiderbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backrub/.

Rule Path
Disallow /

baidu web search

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

big brother

Rule Path
Disallow /

black hole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

botalot

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

cactvs chemistry spider

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

checkbot/x.xx lwp/5.x

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

conceptbot/0.3

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

curious george

Rule Path
Disallow /

cyberpatrol sitecat webbot

Rule Path
Disallow /

cyberspyder/2.1

Rule Path
Disallow /

cybeye.com

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

deweb/1.01

Rule Path
Disallow /

discofinder

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

doc

Rule Path
Disallow /

docomo

Rule Path
Disallow /

download demon

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

dumbot

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

enigmabot

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

exb language crawler

Rule Path
Disallow /

explorersearch

Rule Path
Disallow /

express webpictures

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fetch

Rule Path
Disallow /

fish-search-robot

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

flashget

Rule Path
Disallow /

foobot

Rule Path
Disallow /

freecrawl

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

gcreep/1.0

Rule Path
Disallow /

genieo

Rule Path
Disallow /

genio

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

golem/1.1

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

happyfunbot

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack

Rule Path
Disallow /

httrack [nc,or]

Rule Path
Disallow /

httrack 3.0

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

image stripper

Rule Path
Disallow /

image sucker

Rule Path
Disallow /

inagist.com url crawler

Rule Path
Disallow /

incywincy/1.0b1

Rule Path
Disallow /

indy library

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

informant

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet ninja

Rule Path
Disallow /

israelisearch/1.0

Rule Path
Disallow /

iti spider

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

joc web spider

Rule Path
Disallow /

jubiirobot

Rule Path
Disallow /

jumpstation

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

keyword.density

Rule Path
Disallow /

labelgrab/1.1

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

libcrawl

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

linklooker

Rule Path
Disallow /

linko

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkscan/8.1a unix

Rule Path
Disallow /

lwnutch

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mass downloader

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

merzscope

Rule Path
Disallow /

metagopher

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

midown tool

Rule Path
Disallow /

moget

Rule Path
Disallow /

momspider/1.00 libwww-perl/0.40

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

navroad

Rule Path
Disallow /

neosciocrawler

Rule Path
Disallow /

net vampire

Rule Path
Disallow /

netcarta cyberpilot pro

Rule Path
Disallow /

netzip

Rule Path
Disallow /

netzip downloader 1.0 win32(nov 12 1998)

Rule Path
Disallow /

npbot

Rule Path
Disallow /

occam/1.0

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

open text site crawler v1.0

Rule Path
Disallow /

openbot

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

papa foto

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

radiation retriever 1.1

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

rediffnewsbot

Rule Path
Disallow /

reget

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

repparser

Rule Path
Disallow /

resume robot

Rule Path
Disallow /

rma

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

roverbot

Rule Path
Disallow /

safetynet robot 0.1

Rule Path
Disallow /

scanmine

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

senrigan/xxxxxx

Rule Path
Disallow /

seoengworldbot

Rule Path
Disallow /

seolyticscrawler/3.0

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

seoscanners.net/1

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

sindice-site-manager

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

snooper/b97_01

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

solbot/1.0 lwp/5.07

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spyder3.microsys.com

Rule Path
Disallow /

sqworm/2.9.85-beta (beta_release; 20011115-775; i686-pc-linux

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

url control

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

urlspiderpro

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

vbseo

Rule Path
Disallow /

voideye

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

web sucker

Rule Path
Disallow /

web.image.collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcapture 2.0

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcopier v.2.2

Rule Path
Disallow /

webcopier v3.2a

Rule Path
Disallow /

webcopy/

Rule Path
Disallow /

webcrawler/3.0 robot libwww/5.0a

Rule Path
Disallow /

webdatacentrebot

Rule Path
Disallow /

webemailextrac.*

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webferret

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webgo is

Rule Path
Disallow /

weblayers/0.0

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

weblinker/0.0 libwww-perl/0.1

Rule Path
Disallow /

webmasterworld extractor

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website extractor

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwatch

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

widow

Rule Path
Disallow /

wocbot

Rule Path
Disallow /

woriobot

Rule Path
Disallow /

www collector

Rule Path
Disallow /

www.freeloader.com.

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

xaldon webspider

Rule Path
Disallow /

xenu

Rule Path
Disallow /

xenu\\\'s

Rule Path
Disallow /

xenu\\\'s link sleuth 1.1c

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xget/0.7

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yahoo! slurp china

Rule Path
Disallow /

yolinkbot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

zao

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zeus link scout

Rule Path
Disallow /

zibber-v0.1(www.zibb.com/crawler/)

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

zyborg

Rule Path
Disallow /
Back to top

Comments

  • Bloqueo de trackbacks
  • Bloqueo de feeds para crawlers
  • Bloqueo de bots y crawlers para foros
  • Previene problemas de recursos bloqueados en Google Webmaster Tools (GWT)
  • ==============================================================================
  • ================================ SPECIFIC BOTS SETTINGS =====================
  • robots de Posicionamiento-seo
  • Configuración sacada de PcComponentes.com, Seotecnico.com y https://www.casabatllo.es/
  • ==============================================================================
  • Bots y crawlers de Navegadores admitidos
  • GOOGLE
  • BING y YAHOO ===================
  • Bingbot.: El crawler estándar encargado de rastrear e indexar los sitios
  • MSNBOT.: Este bot solía ser el crawler estándar antes de BingBot, pero aún maneja algunas de las tareas de crawling.
  • MSNBot-Media.: Es el rastreador que maneja imágenes y videos a través de los sitios rastreados del motor de búsqueda Bing.
  • AdIdxBot.: Representa el rastreador utilizado por Bing Ads para rastrear los anuncios y hacer un seguimiento de los sitios web enlazados desde esos anuncios con fines de control de calidad.
  • BingPreview.: Representa el nombre del crawler utilizado para generar instantáneas de la página.
  • YANDEX =====================================
  • Bots y crawlers admitidos temporalmente
  • User-agent: Screaming Frog SEO Spider
  • Disallow: /
  • User-agent: Yeti
  • Disallow: /
  • Ralentizamos algunos bots que se suelen volver locos
  • ========================================================================================================================
  • Bloqueo basico para todos los bots y crawlers
  • Disallow not allowed robots
  • ========================================================================================================================
Back to top

Warnings

  • 5 invalid lines.
  • `visit-time` is not a known field.
Back to top