Log In Sign Up

adrisanhawks.com
robots.txt

Robots Exclusion Standard data for adrisanhawks.com

Resource Scan

Scan Details

Site Domain adrisanhawks.com
Base Domain adrisanhawks.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2026-01-05T22:22:10+00:00
Next Scan 2026-03-06T22:22:10+00:00

Last Successful Scan

Scanned2025-11-06T20:22:50+00:00
URL https://adrisanhawks.com/robots.txt
Domain IPs 213.158.84.33
Response IP 213.158.84.33
Found Yes
Hash 5c7d3a804cde7e0865ca4ad29b460170bc3be871ddcabdb27805e1e131da0805
SimHash 6af4dc800552

Groups

*

Rule Path
Allow /wp-content/uploads/
Disallow /cgi-bin
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-includes/
Disallow /wp-admin/
Disallow /wp-
Disallow /?s=
Disallow /search
Allow /feed/$
Disallow /feed
Disallow /comments/feed
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$
Allow /*.js$
Allow /*.css$

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap http://adrisanhawks/sitemap_index.xml
Back to top

Comments

  • Desindexar carpetas que empiecen por wp-
  • Permitir sitemap pero no las búsquedas.
  • Permitir Feed general para Google Blogsearch.
  • Impedir que /permalink/feed/ sea indexado pues el feed de comentarios suele posicionarse antes de los post.
  • Impedir URLs terminadas en /trackback/ que sirven como Trackback URI (contenido duplicado).
  • Evita bloqueos de CSS y JS.
  • Lista de bots que deberías permitir.
  • Lista de bots que generan consultas abusivas aunque siguen las pautas del archivo robots.txt
  • Slurp (Yahoo!), Noxtrum y el bot de MSN que suelen generar excesivas consultas.
Back to top