Log In Sign Up

revista-gadget.es
robots.txt

Robots Exclusion Standard data for revista-gadget.es

Resource Scan

Scan Details

Site Domain revista-gadget.es
Base Domain revista-gadget.es
Scan Status Ok
Last Scan2024-06-21T06:29:25+00:00
Next Scan 2024-06-28T06:29:25+00:00

Last Scan

Scanned2024-06-21T06:29:25+00:00
URL https://revista-gadget.es/robots.txt
Redirect https://www.revista-gadget.es/robots.txt
Redirect Domain www.revista-gadget.es
Redirect Base revista-gadget.es
Domain IPs 149.202.162.106
Redirect IPs 149.202.162.106
Response IP 149.202.162.106
Found Yes
Hash ceeffaa8dc880f219e1347eaf66fe12dbf01ff53b03279fe10025e0988f5b097
SimHash eaf4d9800752

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /wp-admin/
Disallow /downloads/
Disallow /rayban
Disallow /plesk-stat/
Allow /feed/$
Allow /*.js$
Allow /*.css$
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Disallow: /wp-content/plugins/
  • Disallow: /wp-content/themes/
  • Disallow: /wp-includes/
  • Disallow: /author/
  • Allow: /author/
  • Disallow: /author/*/page/*
  • Disallow: /wp-
  • Permitir Feed general para Google Blogsearch.
  • Impedir que /permalink/feed/ sea indexado pues el feed de comentarios suele posicionarse antes de los post.
  • Impedir URLs terminadas en /trackback/ que sirven como Trackback URI (contenido duplicado).
  • Disallow: /feed
  • Disallow: /comments/feed
  • Disallow: /*/feed/$
  • Disallow: /*/feed/rss/$
  • Disallow: /*/trackback/$
  • Disallow: /*/*/feed/$
  • Disallow: /*/*/feed/rss/$
  • Disallow: /*/*/trackback/$
  • Disallow: /*/*/*/feed/$
  • Disallow: /*/*/*/feed/rss/$
  • Disallow: /*/*/*/trackback/$
  • Evita bloqueos de CSS y JS.
  • Lista de bots que deberías permitir.
  • Lista de bots que generan consultas abusivas aunque siguen las pautas del archivo robots.txt
  • Slurp (Yahoo!), Noxtrum y el bot de MSN que suelen generar excesivas consultas.
Back to top