Log In Sign Up

alquimiarockclub.com.br
robots.txt

Robots Exclusion Standard data for alquimiarockclub.com.br

Resource Scan

Scan Details

Site Domain alquimiarockclub.com.br
Base Domain alquimiarockclub.com.br
Scan Status Ok
Last Scan2024-09-20T04:49:33+00:00
Next Scan 2024-09-27T04:49:33+00:00

Last Scan

Scanned2024-09-20T04:49:33+00:00
URL https://alquimiarockclub.com.br/robots.txt
Domain IPs 108.179.253.73
Response IP 108.179.253.73
Found Yes
Hash ec83293ae080c0be709d7154209945ff2ec9da7e1cc38e838f218b5df2bae687
SimHash 26d05b908776

Groups

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

facebookexternalhit/1.1

Rule Path
Allow /

facebot

Rule Path
Allow /

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap http://www.alquimiarockclub.com.br/sitemap.xml
Back to top

Comments

  • robots.txt para un blog WordPress.
  • Permitir sitemap pero no las búsquedas.
  • Evita bloqueos de CSS y JS.
  • Lista de bots que deberías permitir.
  • Lista de bots que generan consultas abusivas aunque siguen las pautas del archivo robots.txt
  • Slurp (Yahoo!), Noxtrum y el bot de MSN que suelen generar excesivas consultas.
Back to top

Warnings

  • 4 invalid lines.
Back to top