Log In Sign Up

cornella.cat
robots.txt

Robots Exclusion Standard data for cornella.cat

Resource Scan

Scan Details

Site Domain cornella.cat
Base Domain cornella.cat
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2025-08-01T13:16:55+00:00
Next Scan 2025-10-30T13:16:55+00:00

Last Successful Scan

Scanned2024-12-12T13:05:46+00:00
URL https://cornella.cat/robots.txt
Redirect https://www.cornella.cat/robots.txt
Redirect Domain www.cornella.cat
Redirect Base cornella.cat
Domain IPs 137.135.184.158
Redirect IPs 137.135.184.158
Response IP 137.135.184.158
Found Yes
Hash 0f102a1a702c90e48170b3ac8f9979018d0b09c2014e27b41093601cedc81c77
SimHash ac51ab554d61

Groups

*

Rule Path
Disallow /files/
Disallow /ca/portada/
Disallow /es/portada/
Disallow /ca/media/
Disallow /es/media/

googlebot

Rule Path
Disallow /*?
Disallow /*atct_album_view$
Disallow /*folder_factories$
Disallow /*folder_summary_view$
Disallow /*login_form$
Disallow /*mail_password_form$
Disallow /*search
Disallow /*search_rss$
Disallow /*sendto_form$
Disallow /*summary_view$
Disallow /*thumbnail_view$
Disallow /*view$
Back to top

Other Records

Field Value
sitemap https://www.cornella.cat/sitemap.xml.gz
Back to top

Comments

  • Define access-restrictions for robots/spiders
  • http://www.robotstxt.org/wc/norobots.html
  • By default we allow robots to access all areas of our site
  • already accessible to anonymous users
  • Add Googlebot-specific syntax extension to exclude forms
  • that are repeated for each piece of content in the site
  • the wildcard is only supported by Googlebot
  • http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling
Back to top