Aptly. How to organize control of packages from external repositories and delegate control to product teams
Nowadays, many companies operate without the ability to directly control the composition of packages in external repositories, even if they use mirroring, proxying and caching. This leads to the fact that the execution environment is constantly changing, in particular, the composition of docker images changes more often than production requires.
There are situations when unwanted changes contained in external dependencies can get into the composition of the product being developed. This is especially true during product certification. As a result, certification delays, nighttime tests and integration testing failures, breakdown of on-premise production (a production environment located on the organization’s own resources) when running a hotfix, and so on. In a new article, we have described an approach to avoid these problems.
What we wanted to achieve
Before proceeding with the description of the approach, a few words about the tasks that we wanted to solve:
- Get full control over the composition of external packages in the release (predictability).
- Fix the compositions of external repositories for quick roll-out of hotfixes with minimal additional testing (speed).
- Provide product QA stands with a repeatable, predictable, fixed environment (repeatability).
- Independence from the presence of an external communication channel (autonomy).
- Instant switching to official repositories in case of failure (fault tolerance).
- Guaranteed validation of external repository keys in assembly pipelines (trust).
- And most importantly, transfer management and control over the composition of external packages into the hands of product teams and release managers (self-management).
Feature Build Life Cycle Analysis
Our approach solves the problem of fixing the composition of external repositories for a specific date, for a release or a feature. The following diagram illustrates the management of the release, feature build and hotfix lifecycle.
Let’s take the Debian Stretch conditional repository as an example. This approach is applicable to Docker, SaltStack, etc. repositories. Three slices were recorded on the timeline for the dates T1, T2 and T3.
| T1 | T2 | T3 | |
|---|---|---|---|
| stretch | 20200305 | 20200420 | 20200615 |
| Feature1 | 20200304 | 20200304 | 20200501 |
| Feature2 | 20200304 | 20200304 | 20200601 |
| Feature3 | 20200301 | 20200406 | 20200406 |
We have tabulated the contents of the external Debian Stretch repository for building the Feature1, Feature2 and Feature3 distributions. From the table you can see that the composition of the external repository is controlled by each branch independently. We have made an agreement for ourselves to commit the master branch for Debian Stretch on a daily basis and tag each slice in YYYYMMDD format, for example 2020304 for the March 4th, 2020 slice. The table summarizes the snapshots of the external repository used for the distribution in each branch at three different points in time and the composition in the wizard for Debian Stretch. The team for each feature or for each release updates the composition of external repositories at its discretion and according to its development cycle.
Using Feature1 as an example: the product team starts developing a new feature and fixes the composition of the external repository in the configuration files as of the date 20200228 (see the diagram above).
Switch to 20200228 deb http://repository.co/debian-stretch-20200228 stretch main contrib non-free
During the development process, due to the appearance of new packages, it becomes necessary to update the package database to the date 20200304. Switch the working repository to the desired date.
Switch to 20200304 deb http://repository.co/debian-stretch-20200304 stretch main contrib non-free
Then there is another switch of the packet base on the date 20200501.
Switch to 20200501 deb http://repository.co/debian-stretch-20200501 stretch main contrib non-free
If we now draw time slices, we will see that at time points T1 and T2, Feature1 development is proceeding on the package base, “frozen” on March 4, 2020. And in the T3 cut, development is already underway on a new package base for May 1, 2020.
Product multi-release dependency management
Now let’s look at dependency management for several active product releases. There are three releases on support, 2.5, 2.6 and 2.7.
In the table, we see the correspondence of releases and dates for which the repository snapshot was made. It shows which snapshot of the external repository was used to build a specific version of the distribution.
| Release | Composition |
|---|---|
| 2.5.128, 2.5.135, 2.5.207 | 20200301 |
| 2.6.201, 2.6.215, 2.6.315 | 20200301 |
| 2.7.210, 2.7.217, 2.7.305 | 20200404 |
Instead of naming slices by dates YYYYMMDD, we also use tag naming in the format
Snapshot for 2.5 as of 20200301deb http://repository.co/debian-stretch-projectname-2.5 stretch main contrib non-free # 20200301
Thus, during the development of release 2.5, the team fixes the composition of the dependent repositories as of 20200301. Somewhere in April, the team starts a new 2.6 release and decides to use the composition of the external repository packages from 2.5. Create a new 2.6 snapshot from the 2.5 snapshot. In the future, the repositories for 2.5 and 2.6 releases could easily diverge. We have made our own tag debian-stretch-projectname-2.6 for 2.6.
Snapshot for 2.6 as of 20200301deb http://repository.co/debian-stretch-projectname-2.6 stretch main contrib non-free # 20200301
In the case of release 2.7, the team can start development from the master branch – a daily snapshot of the original repository.
Snapshot for 2.7 as of 20200404 deb http://repository.co/debian-stretch-projectname-2.7 stretch main contrib non-free # 20200404
Multi-product dependency management
Let’s look at multi-product dependency management using the example of two products with different release cycles and their own product teams: Stealth and Infiniti.
Let’s comment on the table what happens and when.
| Product | Release | Composition |
|---|---|---|
| stealth2.2 | r2.2.124 | 20200301 |
| stealth2.2 | r2.2.131, r2.2.162 | 20200305 |
| infiniti4.0 | r4.0.235, r4.0.241 | 20200303 |
| infiniti4.0 | r4.0.250 | 20200308 |
1. Let the development of version 2.2 of the Stealth project start on March 1, 2020, for this a snapshot of the composition of the package database for the current date was created. The release release 2.2.124 is made with the external repository package database from 20200301.
Stealth 2.2deb http://repository.co/debian-stretch-stealth-2.2 stretch main contrib non-free # 20200301
2. On the fifth day, the package database is updated. The debian-stretch-stealth-2.2 working repository switches instantly to the desired date, the release of releases 2.2.131 and 2.2.162 is made with the external repository packages from 20200305. Without additional manipulations in the environment, all 100,500 microservices of the product simultaneously received a new environment in the assembly line 20200305…
Stealth 2.2deb http://repository.co/debian-stretch-stealth-2.2 stretch main contrib non-free # 20200305
3. Parallel to the third day, the development of the Infiniti version 4.0 project starts and a slice of the repository composition for the date 20200303 is created for it. Versions 4.0.235 and 4.0.241 are released with the composition of the external repository packages for 20200303.
Infiniti 4.0 deb http://repository.co/debian-stretch-infiniti-4.0 stretch main contrib non-free # 20200303
4. After the release of version 4.0.241, the team decides to update the composition of the repository to 20200308 and release a new release with a new composition of external packages. Version 4.0.250 is released with packages as of 20200308.
Infiniti 4.0 deb http://repository.co/debian-stretch-infiniti-4.0 stretch main contrib non-free # 20200308
Two options for switching between the states of the repositories allow you to choose an approach that is convenient for the development process. In the first case, we switch to the desired state by specifying a snapshot of the repositories on a specific date. In the second case, for multicomponent products, we use a named slice and move it to the desired date. This mechanism provides a one-time cutoff switching in all 100500 product components.
We manage the slices of each external repository in a separate Docker container, so at any time we can switch a specific repository to download from the external network in the event of any accidents.
Download a list of all repositories
# For example
curl repository.co/info/sources.list | grep $(lsb_release -cs) > /etc/apt/sources.listAutomatic creation of slices of external repositories
The repositories are updated every night according to the GitLab scheduler. When a new repository is added, the changes are automatically applied to the server.
At the time of committing a new slice of the external repository, its certificate is checked, if it differs from the one saved with us, then the update does not occur, and we receive an error message.
Outcome
- Preparing a new version of a distribution kit for certification is no longer a headache. For the certification period, we fix the composition of the distribution, and if you need to fix something promptly, then most likely there will be no errors in the released hotfix due to changes in the environment.
- All feature builds get the managed state of external repositories.
- The rollout of hotfixes and verification through QA are accelerated with a predictable, quick and successful result.
- Feature stands receive a predefined, immutable runtime upon deployment.
- Product teams received full independent control over the composition of external repositories for any tasks.
Note that Debian has an official resource snapshot.debian.org with daily cuts and great storage depth. This is sufficient for certain tasks.
Thanks to Sergey Smirnov and the community for an excellent tool for managing the composition of Aptly’s external repositories. A small contribution from us to the best practices for using this useful tool in production conveyors.
In the next articles, we will talk about the Aptly + Simple-CDD bundle for preparing ISO images of distributions, delegating the management of external dependencies to product teams, and the problems of using Aptly in the process of managing external dependencies.
Authors: Nikita Drachev, Alexander Pazdnikov, Timur Gilmullin
