Log In Sign Up

techdroy.com
robots.txt

Robots Exclusion Standard data for techdroy.com

Resource Scan

Scan Details

Site Domain techdroy.com
Base Domain techdroy.com
Scan Status Ok
Last Scan2024-11-16T14:14:40+00:00
Next Scan 2024-11-23T14:14:40+00:00

Last Scan

Scanned2024-11-16T14:14:40+00:00
URL https://techdroy.com/robots.txt
Domain IPs 104.21.6.225, 172.67.135.106, 2606:4700:3033::6815:6e1, 2606:4700:3037::ac43:876a
Response IP 172.67.135.106
Found Yes
Hash 12107d88a26625c9179f60314dc90caad16e25d58ad3dfc754c4fcc789db9394
SimHash e8dd4a020c34

Groups

*

Rule Path
Allow /wp-content/uploads/*
Allow /wp-content/*.js
Allow /wp-content/*.css
Allow /wp-includes/*.js
Allow /wp-includes/*.css
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-login
Disallow /cgi-bin
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-includes/
Disallow /*/attachment/
Disallow /etiqueta/*/page/
Disallow /etiqueta/*/feed/
Disallow */page/*
Disallow /*?s=
Disallow /?cid=
Disallow /?filter_by=
Disallow /?id_lang=
Disallow /?wptouch_preview_theme=
Disallow /comments/
Disallow /xmlrpc.php
Disallow /?attachment_id*

*

Rule Path
Disallow /trackback
Disallow /*trackback
Disallow /*trackback*
Disallow /*/trackback

*

Rule Path
Allow /feed/$
Disallow */feed/
Disallow /feed/
Disallow /comments/feed/
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

sitecheck.internetseer.com
zealbot
msiecrawler
sitesnagger
webstripper
webcopier
fetch
offline explorer
teleport
teleportpro
webzip
linko
httrack
microsoft.url.control
xenu
larbin
libwww
zyborg
download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

semrushbot
ahrefsbot

Rule Path
Disallow /

googlebot

Rule Path
Allow /*.css$
Allow /*.js$

npbot

Rule Path
Disallow /

webreaper
cncdialer
maxthon
mj12bot
slurp

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://techdroy.com/sitemap_index.xml
sitemap https://techdroy.com/post-sitemap.xml
Back to top

Comments

  • Archivo robots.txt de Techdroy
  • Sitemap
  • Bloqueo basico para todos los bots y crawlers
  • Bloqueo de trackbacks
  • Bloqueo de feeds para crawlers
  • Bloqueo de crawlers poco utiles
  • Bloqueo de bots poco utiles
  • wget en su modo recursivo es un problema frecuente
  • El cliente distribuido 'grub' se ha comportado muy mal
  • No sigue el archivo robots.txt de todos modos, pero...
  • Adiós páginas de estadísticas
  • Previene problemas de recursos bloqueados en Google Webmaster Tools
  • Hace demasiadas llamadas
  • Un bot de captura, descarga miles de millones de páginas sin beneficio público
Back to top