Log In Sign Up

eurekarecetas.lavanguardia.com
robots.txt

Robots Exclusion Standard data for eurekarecetas.lavanguardia.com

Resource Scan

Scan Details

Site Domain eurekarecetas.lavanguardia.com
Base Domain lavanguardia.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-28T14:12:50+00:00
Next Scan 2024-06-26T14:12:50+00:00

Last Successful Scan

Scanned2023-11-07T13:39:24+00:00
URL http://eurekarecetas.lavanguardia.com/robots.txt
Redirect https://eurekarecetas.com/robots.txt
Redirect Domain eurekarecetas.com
Redirect Base eurekarecetas.com
Domain IPs 82.98.165.188
Redirect IPs 82.98.165.188
Response IP 82.98.165.188
Found Yes
Hash 4ee81e92bb8d8c84b51bc0a67382549ada54ec39a269ad5261d6d7ed630e1a54
SimHash b8941f09c778

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

grapeshot

Rule Path
Disallow
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Allow /sites/*.css$
Allow /sites/*.css?
Allow /sites/*.js$
Allow /sites/*.js?
Allow /sites/*.gif
Allow /sites/*.jpg
Allow /sites/*.jpeg
Allow /sites/*.png
Allow /misc/*.css$
Allow /misc/*.css?
Allow /misc/*.js$
Allow /misc/*.js?
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • CSS, JS, Images
  • DYN: inicio
  • DYN: fin
Back to top