Log In Sign Up

gitlab.orchi.tech
robots.txt

Robots Exclusion Standard data for gitlab.orchi.tech

Resource Scan

Scan Details

Site Domain gitlab.orchi.tech
Base Domain orchi.tech
Scan Status Ok
Last Scan2024-11-07T23:01:19+00:00
Next Scan 2024-11-21T23:01:19+00:00

Last Scan

Scanned2024-11-07T23:01:19+00:00
URL https://gitlab.orchi.tech/robots.txt
Domain IPs 217.11.251.148
Response IP 217.11.251.148
Found Yes
Hash 45e0dae6d55c481103bc789e95068005c4b4022eb8a81d85332a8bd484502364
SimHash 06b61c552176

Groups

*

Rule Path
Disallow /autocomplete/users
Disallow /search
Disallow /api
Disallow /admin
Disallow /profile
Disallow /dashboard
Disallow /projects/new
Disallow /groups/new
Disallow /groups/*/edit
Disallow /users

*

Rule Path
Disallow /s/
Disallow /snippets/new
Disallow /snippets/*/edit
Disallow /snippets/*/raw

*

Rule Path
Disallow /*/*.git
Disallow /*/*/fork/new
Disallow /*/*/repository/archive*
Disallow /*/*/activity
Disallow /*/*/new
Disallow /*/*/edit
Disallow /*/*/raw
Disallow /*/*/blame
Disallow /*/*/commits/*/*
Disallow /*/*/commit/*.patch
Disallow /*/*/commit/*.diff
Disallow /*/*/compare
Disallow /*/*/branches/new
Disallow /*/*/tags/new
Disallow /*/*/network
Disallow /*/*/graphs
Disallow /*/*/milestones/new
Disallow /*/*/milestones/*/edit
Disallow /*/*/issues/new
Disallow /*/*/issues/*/edit
Disallow /*/*/merge_requests/new
Disallow /*/*/merge_requests/*.patch
Disallow /*/*/merge_requests/*.diff
Disallow /*/*/merge_requests/*/edit
Disallow /*/*/merge_requests/*/diffs
Disallow /*/*/project_members/import
Disallow /*/*/labels/new
Disallow /*/*/labels/*/edit
Disallow /*/*/wikis/*/edit
Disallow /*/*/snippets/new
Disallow /*/*/snippets/*/edit
Disallow /*/*/snippets/*/raw
Disallow /*/*/deploy_keys
Disallow /*/*/hooks
Disallow /*/*/services
Disallow /*/*/protected_branches
Disallow /*/*/uploads/
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-Agent: *
  • Disallow: /
  • Add a 1 second delay between successive requests to the same server, limits resources used by crawler
  • Only some crawlers respect this setting, e.g. Googlebot does not
  • Crawl-delay: 1
  • Based on details in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/routes.rb, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/routing, and using application
  • Global snippets
  • Project details
Back to top