How the GitLab QA Team Uses the GitLab Performance Tool
We launched a number of initiatives to improve the performance of GitLab and we needed a new benchmark tool.
Performance testing is a complex process that differs from other types of testing. To obtain results, dedicated resources, special approaches and tools are needed here. When I joined the company and became the first member of this team, my task was to transfer the nascent efforts to improve productivity to a larger scale. To do this, we made a new tool with a simple name. Gitlab performance tool (GPT).
We are pleased to announce a public release. GPT. In this post, we will describe how we use GPT to test GitLab performance, and also how you can use it to test your environments.
However, first look at what we use it with.
Reference architectures and test data
In our experience, the most difficult part in performance testing is not in the testing itself, but in setting up the correct environments and test data.
Thus, one of the initiatives is the development of GitLab reference architectures (Gitlab reference architectures) that can handle a large number of users. We want to use these architectures as a way to standardize our recommended configurations to provide customers with high-performance, scalable, and highly accessible GitLab deployments.
For such a tool, we need realistic test data, for example, large projects with commits and merge requests. For the first iteration, we took the project of GitLab’s.
After setting up and running the test environments, we were ready to test them using the GPT.
What is the GitLab Performance tool (GPT)?
GPT can be used to run various load tests for the GitLab benchmark. All that is required is to find out what throughput the test environment can withstand (in the form of requests per second), and ensure the preparation of test data.
GPT is developed on the basis of one of the leading tools for load testing – k6. Here are some of the GPT features:
- A large set of tests out of the box, covering various GitLab endpoints, with the ability to add your own tests. Description of finished tests see in the documentation, new tests are often added there.
- Parameters to run tests, such as the necessary data for the GitLab environment or the bandwidth used.
- Run tests sequentially with a selection of tests to run.
- Advanced reporting and logging.
- Built-in conditions for verifying successful test completionbased on time to first byte (time to first byte), throughput achieved, or successful responses.
Talented team Load impact created k6which is the core of GPT. We quickly realized that we did not need to reinvent the wheel, as k6 satisfies most of our requirements: it is written in Go, which is why it is very productive and open source. Thanks to the team not only for developing k6, but also for working with us.
How do we use GPT
GPT is used in several automated GitLab CI pipelines to provide quick feedback on the operation of GitLab. CI pipelines of reference architectures work with the latest pre-release and, as a rule, run daily or weekly. Immediately after testing, we analyze the results to find problems. All latest results we publish in GPT wiki according to our Transparency value.
GPT is also used in the benchmarking pipeline to see how GitLab’s performance changes from release to release. These results are important because they show a picture of improved performance. results benchmark comparisons also available in GPT wiki.
Using GPT, we were able to identify several bottlenecks in GitLab’s performance and, together with developers, prioritize improvements. This process has paid off, and we look forward to improving performance with every release of GitLab. For example, release 12.6 showed some notable improvements in all directions, in one of which we achieved a decrease of 92%! You can look at the detected problems in our bug tracker.
How can you use GPT
We have long decided that we want to follow the same principles of open source as for our main product, therefore, when creating the GPT, we focused on all users, and not on making it an exclusively internal product. Therefore, we not only allow others to use it, but also encourage it! This is beneficial both for us and our clients, as we receive feedback from different points of view, which we did not think about. Some examples of this are improvement. bandwidth-based recommended spec guides, or improvements in using GPT offline for users with private clouds.
If you want to use GPT for yourself, it is best to start with documentation. As noted, most of the effort to use GPT is to prepare the appropriate environments. The documentation discusses both this point and the direct use of GPT.
GPT in action
Finally, after talking about the GPT, it’s time to show it in action. Here’s what a load test run looks like for List Group Projects API with 10k reference architecture:
asciinema.org/a/O96Wc5fyxvLb1IDyviTwbujj8
For more information on the results, see the documentation GPT.
What’s next
Our goal is to make GitLab, in terms of performance, the best in its class. And we are only at the beginning of this journey. We are pleased that we can improve the quality of customer service by providing them with additional tools.
Some of our plans for future releases include expanding the test coverage of GitLab functionality, entry points (API, Web, Git), expanding work on reference architectures, test data and user behavior patterns to make them more representative and realistic.
Share your reviews and / or suggestions on GPT in the repository of the GPT project! We welcome your ideas and suggestions.
