Log In Sign Up

mlmonferrer.es
robots.txt

Robots Exclusion Standard data for mlmonferrer.es

Resource Scan

Scan Details

Site Domain mlmonferrer.es
Base Domain mlmonferrer.es
Scan Status Ok
Last Scan2025-06-17T13:36:10+00:00
Next Scan 2025-06-24T13:36:10+00:00

Last Scan

Scanned2025-06-17T13:36:10+00:00
URL https://mlmonferrer.es/robots.txt
Domain IPs 2001:8d8:100f:f000::2b4, 217.160.0.151
Response IP 217.160.0.151
Found Yes
Hash be9b5a132446386f8ef25f9670d82a057ec2a1df95cd846977b5fb1e0fa39595
SimHash c844da82ef54

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /comments/
Disallow /archives/
Disallow /author/
Disallow /*?
Disallow /?s=
Disallow /search/
Disallow /*q%3D
Disallow /comments/feed
Allow /feed/$
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$
Disallow /*/function.file-exists$
Back to top

Comments

  • Eliminamos posibles paginas con contenido duplicado
  • Eliminamos cualquier tipo de busqueda y URLS con parametros
  • mas
Back to top