Expanding the menu and functionality of Gitlab
During the rapid development of the IT environment, many of its active participants use ready-made solutions that provide them with certain functionality that they would like to expand. But expanding a product is often either a paid or extremely expensive activity that requires constant monitoring of the available code base, adaptation and adjustment of its part for compatibility. We have to create products placed “side by side” and integrate in various ways. Let's try to change this with minimal costs and expand the functionality of Gitlab by obtaining almost “seamless” integration with our products.
How does Gitlab work?
To understand what and how it works, you need to decide on the topology of Gitlab as a product as a whole. Among the main parts it is worth highlighting:
puma – internal web server, responsible for the web interface and content rendering
gitlab-workhorse – internal reverse-proxy, responsible for processing external user requests, works in conjunction with nginx and puma
gitlab-kas – authentication module
gitaly – is responsible for working with repositories and provides an internal RPC interface for various types of interaction with repositories
registry – is responsible for the internal registry in which containers or packages are stored
sidekiq is responsible for coordinating internal processes and linking them into tasks performed by the scheduler
postgres is responsible for storing operational information
on top of everything is nginx, which brings together all the parts for external user interaction
We will not consider monitoring services, because they are not within our scope of interests
As part of an article devoted to expanding the functionality of Gitlab, we will be interested in the puma and nginx modules.
Let's dwell a little on puma. Puma is a powerful web server written in ruby, using extensive templating capabilities and providing the preparation of user-visible content, while tightly binding further visualization/interactivity using vue.js and css. Templating pages in Gitlab is almost universal and is based on haml templates, so the structure of almost all pages is the same and includes:
basic header styles depending on the theme selected by the user
a settings block for vue.js placed at the top of the page and containing a reference of settings used for interaction with the user
settings block for graphql queries (part of the pages)
menu block, specified as a json object (generated dynamically depending on user rights)
service block for various notifications
a data block in which content is drawn according to the current menu section
Since Gitlab is a constantly evolving product, its appearance changes quite often. At the same time, some of the setup pages for the ce and ee versions are very visually different. In this regard, it becomes unprofitable to use embedding in haml templates, because you need to focus on a specific version of gitlab. Therefore, the most convenient point of implementation is the menu block, which practically does not change from version to version.
Getting into the main menu
The Gitlab menu is a very interesting topic, given that it is completely interactive and a large piece of Javascript code is responsible for its interactivity, although all the preparation for the magic happens inside the ruby library. When conducting my first research into implementation in Gitlab, I tried several options, but many of them resulted in a 502 error in puma and I had to roll back. The Gitlab menu, located on the left, is called a sidebar in the system and has a dedicated block of code for its work. In new versions of Gitlab, an additional menu has appeared on the right; we won’t talk about it today. Below in the text of the article the method of customizing the menu within the framework of the created extension will be described.
The main part of the menu is defined by the sidebar library located along the path /opt/gitlab/embedded/service/gitlab-rails/lib/sidebars. In fact, all parts of the menu are constructors with a set of parameters responsible for the visual design of the menu item.
Gitlab has many different menu options, but the main one that works for most is “Your Work”. This is what we will talk about next.
In terms of ruby code, the “Your Work” menu is a constructor that describes the topology of menu items located in the file /opt/gitlab/embedded/service/gitlab-rails/lib/sidebars/your_work/panel.rb
# frozen_string_literal: true
module Sidebars
module YourWork
class Panel < ::Sidebars::Panel
override :configure_menus
def configure_menus
add_menus
end
override :aria_label
def aria_label
_('Your work')
end
override :super_sidebar_context_header
def super_sidebar_context_header
aria_label
end
private
def add_menus
return unless context.current_user
add_menu(Sidebars::YourWork::Menus::ProjectsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::GroupsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::OrganizationsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::IssuesMenu.new(context))
add_menu(Sidebars::YourWork::Menus::MergeRequestsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::TodosMenu.new(context))
add_menu(Sidebars::YourWork::Menus::MilestonesMenu.new(context))
add_menu(Sidebars::YourWork::Menus::SnippetsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::ActivityMenu.new(context))
end
end
end
end
Sidebars::YourWork::Panel.prepend_mod_with('Sidebars::YourWork::Panel')
where in turn the Projects menu item is described as
# frozen_string_literal: true
module Sidebars
module YourWork
module Menus
class ProjectsMenu < ::Sidebars::Menu
override :link
def link
dashboard_projects_path
end
override :title
def title
_('Projects')
end
override :sprite_icon
def sprite_icon
'project'
end
override :render?
def render?
!!context.current_user
end
override :active_routes
def active_routes
{ controller: ['root', 'projects', 'dashboard/projects'] }
end
end
end
end
end
As we can see, the format is quite simple, but creating a new file with a description of our menu and implementing it with each update can end sadly. Therefore, you can use a short form to implement it in the menu.
add_menu(::Sidebars::MenuItem.new(title: _('Helper Menu'), link: '/-/helper', sprite_icon: 'text-description', active_routes: {}, super_sidebar_parent: ::Sidebars::YourWork, item_id: :helper))As a result, the file with the “Your Work” menu will look like
# frozen_string_literal: true
module Sidebars
module YourWork
class Panel < ::Sidebars::Panel
override :configure_menus
def configure_menus
add_menus
end
override :aria_label
def aria_label
_('Your work')
end
override :super_sidebar_context_header
def super_sidebar_context_header
aria_label
end
private
def add_menus
return unless context.current_user
add_menu(::Sidebars::MenuItem.new(title: _('Helper Menu'), link: '/-/helper', sprite_icon: 'text-description', active_routes: {}, super_sidebar_parent: ::Sidebars::YourWork, item_id: :helper))
add_menu(Sidebars::YourWork::Menus::ProjectsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::GroupsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::OrganizationsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::IssuesMenu.new(context))
add_menu(Sidebars::YourWork::Menus::MergeRequestsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::TodosMenu.new(context))
add_menu(Sidebars::YourWork::Menus::MilestonesMenu.new(context))
add_menu(Sidebars::YourWork::Menus::SnippetsMenu.new(context))
add_menu(Sidebars::YourWork::Menus::ActivityMenu.new(context))
end
end
end
end
Sidebars::YourWork::Panel.prepend_mod_with('Sidebars::YourWork::Panel')
You can add a personal “icon” to a new menu item by taking its catalog name https://gitlab-org.gitlab.io/gitlab-svgs/. After the command gitlab-ctl restart puma we will see this new type of menu.
How to make menu changes permanent?
The question arises: “Well, I corrected this menu, and at the next update I should edit it again?” To answer this question, there is a “magic” functionality of triggers, using which we will always have our menu item in this file.
While studying the structure and principle of working with triggers, I came across a fairly simple example on github https://github.com/Animalcule/triggered-edit-deb-package considering which the implementation method becomes clear.
To create a trigger, we need to describe the point that will be controlled when installing/updating the package
interest /opt/gitlab/embedded/service/gitlab-rails/lib/sidebars/your_work/panel.rbNext, we need to describe the script that will be called when an event occurs with the file we are interested in.
#!/bin/sh
set -eu
if [ "$1" = "triggered" ]; then
if [ "$2" = "/opt/gitlab/embedded/service/gitlab-rails/lib/sidebars/your_work/panel.rb" ]; then
logger "Fix gitlab 'Your Work' menu"
line="return unless context.current_user"
addline="add_menu(::Sidebars::MenuItem.new(title: _('Helper Menu'), link: '/-/helper', sprite_icon: 'text-description', active_routes: {}, super_sidebar_parent: ::Sidebars::YourWork, item_id: :helper))"
sed -i -e "/$line$/a"'\\n'"\t$addline" "$2"
fi
fi
A deb package is assembled from a set of our scripts and installed on the system. After that, the next time you replace the file panel.rb to the original/new one, our trigger will work and add the required item to the Gitlab menu. The modified source code for generating the package and the package itself can be downloaded from the repository https://github.com/aborche/gitlab-menu-changer/
Initially, it is assumed that the menu item will be added before calling the Gitlab setup procedure and starting the puma service. To test the script, you can simply call the trigger executable file with parameters.
sh /var/lib/dpkg/info/gitlab-menu-changer.postinst triggered /opt/gitlab/embedded/service/gitlab-rails/lib/sidebars/your_work/panel.rbAfter the changes you need to run the command gitlab-ctl restart puma
What's next ?
We seem to have sorted out the menu, but the question arises: “What does this give us? When we click on the menu link, we get a 404/502 error, nothing works!”
For those who did not quite understand what we did in the first part, I will explain. We have created a new main menu item with a link located in the tree of the main Gitlab site, to which all the rules for working in the site tree apply. Local storage, cookies, csrf tokens, etc.
This is where the second part of the “magic” begins. At the very beginning of the article there was a remark about Nginx, which is used to connect all Gitlab services when working with the user. Few people have thought about the role Nginx plays in the Gitlab ecosystem, simply taking its presence for granted. In file /etc/gitlab/gitlab.rb there are a couple of commented lines
# nginx['custom_gitlab_server_config'] = "location ^~ /foo-namespace/bar-project/raw/ {\n deny all;\n}\n"
# nginx['custom_nginx_config'] = "include /etc/nginx/conf.d/example.conf;"The first describes blocking part of the content, the second allows you to add additional config for the http block. This is the key to our Gitlab extension. To display our content it is necessary in the block nginx['custom_gitlab_server_config'] describe the implementation of our location
nginx['custom_gitlab_server_config'] = "include /etc/nginx/gitlab/helper.conf;"and create a file with location
location /-/helper {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Original-ARGS $args;
proxy_set_header X-Remote-Addr $remote_addr;
proxy_set_header X-Original-Host $host;
proxy_pass http://host.with.menu.app:8099;
}
location = /api/v4/graphql {
rewrite /api/v4/graphql /api/graphql last;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_read_timeout 900;
proxy_cache off;
proxy_buffering off;
proxy_request_buffering off;
proxy_http_version 1.1;
proxy_pass http://gitlab-workhorse;
}
After change /etc/gitlab/gitlab.rb must be done gitlab-ctl reconfigure nginx. When further setting up the file with location, you can use the direct restart of nginx with the command gitlab-ctl restart nginx. In this case, you can check the nginx config with the command /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx/ -tT
The config contains a piece for working with graphql in the /api/v4 branch; it is necessary for use within the extension functionality, because the basic functionality of the go-gitlab library (more about it below) is “tailored” to the “/api/v4” prefix and does not allow executing a request outside of this prefix.
Creating our extension
As previously written, by integrating into the Gitlab site tree we receive all the privileges and the entire environment for Gitlab pages. The first extension options were built as static pages using bootstrap, which simply rendered content by calling the api from ajax requests. But I wanted more and decided to inherit the Gitlab page format. In any case, the Gitlab template engine renders pages almost from scratch. Yes, there is a small cache, but it only works for 3-4 page refreshes. Therefore, calling a Gitlab page from a third-party application, passing the necessary cookies and headers, will be no different from working from a browser. As a result, a middleware server was written whose main task was to reformat the Gitlab page into the required format. Go 1.21 was chosen as the development language, because… firstly, I liked the Gin framework, and secondly, I wanted to compare the convenience of Go in terms of processing *ML structures compared to python, java. By the way, it will be said that a very good module for working with the Gitlab api was written in Go https://github.com/xanzy/go-gitlabwhich really had to be corrected a little to work with cookies.
Let's start with the requirements, what is needed for our middleware service and what are our limitations?
The middleware service should provide the following functionality:
Making a request to the Gitlab base/start page to get a nugget
Handle authentication and redirect errors for entering authentication data
Be able to parse the resulting cast and modify it into an internal template
Enrich the resulting template with a new menu, as well as generate the necessary content for visualization
This gives rise to some limitations that must be taken into account:
Contextual cookies. Must be read, processed, enriched and returned to the current user session at all times. More on this later.
Processing, parsing and template generation can negatively affect performance. This should be taken for granted. Everyone should understand that we do not work inside the Gitlab realtime engine, but are engaged in visual integration into the interface.
Any page has a settings block for Vue.js, so the internal templates from which our content display will be built must take this settings block into account.
Rendering elements on a page may not be exactly the same as Gitlab's rendering. This is due to the fact that all interactivity on the page is working with dynamic dom objects using Vue.js scripts and repeating all this is more expensive. There will be more information about this later
Using the Gitlab API is possible in full, with the exception of some methods, for example, “generating a deploy token,” which uses graphql + csrf. This needs to be remembered
Based on the above, we will begin to model the technology of interaction between middleware and Gitlab.
Session cookies
Perhaps one of the most important parts of middleware. As part of the Gitlab session interaction with the user, a different set of cookies is used, and not all of this set is actually needed for middleware. What exactly we need:
known_sign_in – a cookie set at login, on the basis of which Gitlab “understands” whether there was a previous login from the specified machine or not
_gitlab_session – the main session cookie set upon successful login to Gitlab. If there are no cookies, the user is sent to the login/password entry page
remember_user_token – a token generated when using the “Remember me” checkbox, participating in the recreation of a session cookie “_gitlab_session” in its absence
What do you need to remember?
Cookie received by middleware after transfer “remember_user_token” in Gitlab, must be saved in the middleware session when working with the Gitlab API and then exposed to the browser when displaying the requested content
Expired cookie”_gitlab_session“can be automatically replaced with a new one when used “remember_user_token”which will also require performing the actions from step 1
Using graphql with expired cookies on multiple requests does not enable authorization error mode. As a result, many queries simply return an empty data set. Therefore, when working with graphql, double processing of the cookie and subsequent processing of the csrf token is required.
If there is no session cookie “_gitlab_session” Gitlab sends the response “Status:302” to the browser and the middleware should be able to “catch” this moment.
Gitlab page parser
When choosing a parser for processing Gitlab pages used for a middleware template, it should be taken into account that the output of the parser should return a modified template, ready to implement the content we need. In 99% of cases you will be dealing with a dom parser that does not take into account the path to document elements. Therefore, some of the functionality will need to be adapted
What and how should be parsed?
Settings block. Located at the very beginning of the document in the block <script>, and the only identifying mark of the block is that it begins with “window.gon={};”. Therefore, we need a filter by elements
