Log In Sign Up

gitlab.ow2.org
robots.txt

Robots Exclusion Standard data for gitlab.ow2.org

Resource Scan

Scan Details

Site Domain gitlab.ow2.org
Base Domain ow2.org
Scan Status Ok
Last Scan2025-08-04T01:56:55+00:00
Next Scan 2025-09-03T01:56:55+00:00

Last Scan

Scanned2025-08-04T01:56:55+00:00
URL https://gitlab.ow2.org/robots.txt
Redirect https://www.ow2.org/download/Main/WebHome/robots.txt
Redirect Domain www.ow2.org
Redirect Base ow2.org
Domain IPs 54.38.13.219
Redirect IPs 54.38.13.219
Response IP 54.38.13.219
Found Yes
Hash 5cadf3bb41f1f91db96d7eb7b502ce2e2a6c13d0b97ecc0eeec184c8d5bddad7
SimHash b428c9f58ea5

Groups

semrushbot
siteauditbot
ahrefsbot
coherencebot
deskyobot
magpie-crawler
mauibot
coccocbot-image
coccocbot-web
dotbot
infotigerbot
mail.ru_bot
mj12bot
seznambot
surdotlybot
wellknownbot
yandexbot
dataforseobot
sabsimbot
trendictionbot
yacybot
zoominfobot

Rule Path
Disallow /

*

Rule Path
Disallow /view/services/
Disallow /view/Membership_Joining/On_Line_Registration
Disallow /services/
Disallow /status/
Disallow /xmlrpc/
Disallow /view/XWiki

*

Rule Path
Disallow /viewattachrev/
Disallow /viewrev/
Disallow /pdf/
Disallow /tex/
Disallow /edit/
Disallow /create/
Disallow /inline/
Disallow /preview/
Disallow /save/
Disallow /saveandcontinue/
Disallow /rollback/
Disallow /deleteversions/
Disallow /cancel/
Disallow /delete/
Disallow /deletespace/
Disallow /undelete/
Disallow /reset/
Disallow /register/
Disallow /propupdate/
Disallow /propadd/
Disallow /propdisable/
Disallow /propenable/
Disallow /propdelete/
Disallow /objectadd/
Disallow /commentadd/
Disallow /commentsave/
Disallow /objectsync/
Disallow /objectremove/
Disallow /attach/
Disallow /upload/
Disallow /download/
Disallow /temp/
Disallow /downloadrev/
Disallow /dot/
Disallow /svg/
Disallow /delattachment/
Disallow /login/
Disallow /loginsubmit/
Disallow /loginerror/
Disallow /logout/
Disallow /charting/
Disallow /lock/
Disallow /redirect/
Disallow /admin/
Disallow /export/
Disallow /import/
Disallow /get/
Disallow /distribution/
Disallow /imagecaptcha/
Disallow /unknown/
Disallow /view/Sandbox/
Disallow /view/Admin/
Disallow /view/Stats/
Disallow /view/Panels/
Disallow /Main/Search
Disallow /xwiki/rest/
Back to top

Comments

  • Disallow all the website to undesirable bots
  • Syntax reference: https://developers.google.com/search/docs/advanced/robots/create-robots-txt
  • OW2 Custom
  • XWIKI recommendations
  • https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Performances/#HRobots.txt
  • Prevent bots from executing all actions except "view" since:
  • 1) we don't want bots to execute stuff in the wiki!
  • 2) we don't want bots to consume CPU and memory
  • (for example to perform exports)
  • Don't index sandbox content since it's sample content
  • Don't index Admin space since it contains Admin stuff.
  • Note that the Admin space is protected by permissions
  • anyway but this acts as a safety net to not have private
  • info leaked on the internet ;)
  • Don't index Stats data (just because it's not useful and
  • those pages are a bit CPU intensive)
  • Don't index Panels data (because we don't want it
  • indexed on the internet)
  • Don't index the search page.
  • Don't index the REST API.
  • These are just UI elements which can cause infinite loops in
  • web crawlers. See https://jira.xwiki.org/browse/XWIKI-16915
Back to top

Warnings

  • 1 invalid line.
Back to top