Log In Sign Up

cruzcampo.es
robots.txt

Robots Exclusion Standard data for cruzcampo.es

Resource Scan

Scan Details

Site Domain cruzcampo.es
Base Domain cruzcampo.es
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-22T05:11:57+00:00
Next Scan 2024-06-21T05:11:57+00:00

Last Successful Scan

Scanned2024-01-31T01:36:42+00:00
URL https://www.cruzcampo.es/robots.txt
Domain IPs 13.224.249.37, 13.224.249.48, 13.224.249.61, 13.224.249.90
Response IP 13.32.27.51
Found Yes
Hash b1ac4718b38b7e83a144c71f1d7e7b1d4568b36530c643fa9cd000823ef9c939
SimHash 3a591a91ccf1

Groups

*

Rule Path
Disallow /prensa/
Disallow /sostenibilidad/
Disallow /politica-cookies/
Disallow /contacto/
Disallow /politica-privacidad/
Disallow /politica-redes-sociales/
Disallow /aviso-legal/
Disallow /anuncio-cruzcampo/gitana/prensa/

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /*.css$
Allow /*.js$

ahrefsbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

urlappendbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

xenu

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

badbot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

arachnophilia

Rule Path
Disallow /

webcopy

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

israbot

Rule Path
Disallow

orthogaffe

Rule Path
Disallow
Back to top

Comments

  • Bloqueo de urls no index
  • Control ppto rastreo
  • Previene problemas de recursos bloqueados en Google Search Console
  • Bloqueo de cotillas
  • User-agent: SemrushBot
  • Disallow: /
  • User-agent: SemrushBot-SA
  • Disallow: /
  • Bloqueo de bots que no son útiles
  • Bots que clonan el sitio web
  • Wikipedia work bots:
  • Sitemaps
Back to top

Warnings

  • 2 invalid lines.
Back to top