Log In Sign Up

gitlab.fi.muni.cz
robots.txt

Robots Exclusion Standard data for gitlab.fi.muni.cz

Resource Scan

Scan Details

Site Domain gitlab.fi.muni.cz
Base Domain muni.cz
Scan Status Ok
Last Scan2024-06-23T14:25:56+00:00
Next Scan 2024-07-07T14:25:56+00:00

Last Scan

Scanned2024-06-23T14:25:56+00:00
URL https://gitlab.fi.muni.cz/robots.txt
Domain IPs 147.251.58.80, 2001:718:801:23a::50
Response IP 147.251.58.80
Found Yes
Hash d36f8aa5b12fcdccdb8f4028cbb9a2c1a44832dce2f872603b50911536a44bcc
SimHash 06b2395f2976

Groups

*

Rule Path
Disallow /autocomplete/users
Disallow /search
Disallow /api
Disallow /admin
Disallow /profile
Disallow /dashboard
Disallow /projects/new
Disallow /groups/new
Disallow /groups/*/edit
Disallow /users
Disallow /help
Allow /users/sign_in

*

Rule Path
Disallow /s/
Disallow /snippets/new
Disallow /snippets/*/edit
Disallow /snippets/*/raw

*

Rule Path
Disallow /*/*.git
Disallow /*/*/fork/new
Disallow /*/*/repository/archive*
Disallow /*/*/activity
Disallow /*/*/new
Disallow /*/*/edit
Disallow /*/*/raw
Disallow /*/*/blame
Disallow /*/*/commits/*/*
Disallow /*/*/commit/*.patch
Disallow /*/*/commit/*.diff
Disallow /*/*/compare
Disallow /*/*/branches/new
Disallow /*/*/tags/new
Disallow /*/*/network
Disallow /*/*/graphs
Disallow /*/*/milestones/new
Disallow /*/*/milestones/*/edit
Disallow /*/*/issues/new
Disallow /*/*/issues/*/edit
Disallow /*/*/merge_requests/new
Disallow /*/*/merge_requests/*.patch
Disallow /*/*/merge_requests/*.diff
Disallow /*/*/merge_requests/*/edit
Disallow /*/*/merge_requests/*/diffs
Disallow /*/*/project_members/import
Disallow /*/*/labels/new
Disallow /*/*/labels/*/edit
Disallow /*/*/wikis/*/edit
Disallow /*/*/snippets/new
Disallow /*/*/snippets/*/edit
Disallow /*/*/snippets/*/raw
Disallow /*/*/deploy_keys
Disallow /*/*/hooks
Disallow /*/*/services
Disallow /*/*/protected_branches
Disallow /*/*/uploads/
Disallow /*/-/group_members
Disallow /*/project_members

*

Rule Path
Disallow /xmelkov/bitka-pri-mohaci/*$
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-Agent: *
  • Disallow: /
  • Add a 1 second delay between successive requests to the same server, limits resources used by crawler
  • Only some crawlers respect this setting, e.g. Googlebot does not
  • Crawl-delay: 1
  • Based on details in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/routes.rb, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/routing, and using application
  • Only specifically allow the Sign In page to avoid very ugly search results
  • Global snippets
  • Project details
  • Melkovic
Back to top