Log In Sign Up

sfia-online.org
robots.txt

Robots Exclusion Standard data for sfia-online.org

Resource Scan

Scan Details

Site Domain sfia-online.org
Base Domain sfia-online.org
Scan Status Ok
Last Scan2025-09-11T16:40:23+00:00
Next Scan 2025-10-11T16:40:23+00:00

Last Scan

Scanned2025-09-11T16:40:23+00:00
URL https://sfia-online.org/robots.txt
Domain IPs 104.26.0.169, 104.26.1.169, 172.67.75.238, 2606:4700:20::681a:1a9, 2606:4700:20::681a:a9, 2606:4700:20::ac43:4bee
Response IP 104.26.0.169
Found Yes
Hash 03a138805cce9c940ff8801b6d30e15235c3ec8ed109a8395d249b2a8fe945f7
SimHash af71195565e5

Groups

*

Rule Path
Disallow /staging/
Disallow /ar/%40%40search
Disallow /de/%40%40search
Disallow /en/%40%40search
Disallow /en-us/%40%40search
Disallow /es/%40%40search
Disallow /fr/%40%40search
Disallow /fr-ca/%40%40search
Disallow /it/%40%40search
Disallow /ja/%40%40search
Disallow /nl/%40%40search
Disallow /pt/%40%40search
Disallow /pt/%40%40search
Disallow /ru/%40%40search
Disallow /zh/%40%40search
Disallow /ar/search_rss
Disallow /de/search_rss
Disallow /en/search_rss
Disallow /en-us/search_rss
Disallow /es/search_rss
Disallow /fr/search_rss
Disallow /fr-ca/search_rss
Disallow /it/search_rss
Disallow /ja/search_rss
Disallow /nl/search_rss
Disallow /pt/search_rss
Disallow /pt/search_rss
Disallow /ru/search_rss
Disallow /zh/search_rss

googlebot

Rule Path
Disallow /*atct_album_view$
Disallow /*folder_factories$
Disallow /*folder_summary_view$
Disallow /*login_form$
Disallow /*mail_password_form$
Disallow /%40%40search
Disallow /*search_rss$
Disallow /*sendto_form$
Disallow /*summary_view$
Disallow /*thumbnail_view$
Disallow /*view$

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://sfia-online.org/sitemap.xml.gz
Back to top

Comments

  • For the default, see ROBOTS_TXT in https://github.com/plone/plone.base/blob/main/src/plone/base/interfaces/controlpanel.py
  • Define access-restrictions for robots/spiders
  • http://www.robotstxt.org/wc/norobots.html
  • By default we allow robots to access all areas of our site
  • already accessible to anonymous users
  • Disallow the search form on all languages: all content should be reachable via other links or the sitemap.
  • Also disallow the search_rss page on all languages, for the same reason.
  • Add Googlebot-specific syntax extension to exclude forms
  • that are repeated for each piece of content in the site
  • the wildcard is only supported by Googlebot
  • http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling
  • https://bot.seekport.com
Back to top