Preparing private repositories with Artipie

Numerous sanctions restrictions can become an obstacle to the use of existing binary repository management systems, for example, when acquiring the necessary licenses. Thus, an alternative to existing tools can play a positive role in organizing work with organizational artifacts.

Artipie is a free and open source binary repository manager that supports all major package and artifact formats, allowing you to host them locally or in the cloud.

Binary spins
Binary spins

Introduction

The Binary Repository Manager (MBR) is a software tool designed to optimize the download, upload and storage of binary files used and produced in the context of software development. As one of the key components of DevOps tools, the MBR is located right after the build pipeline. That is why it is sometimes called “assembly store”, “artifact store” or “pipeline state store”.

The traditional DevOps pipeline expects source code, which is automatically verified, tested, packaged, and versioned into an artifact (binary file) such as .jar, .tar, .zip, or other formats. The artifact is then typically stored outside of the source code repository and must be available to subsequent stages of the Continuous Delivery pipeline. As a central point for managing binaries and their dependencies, and as an integrated repository for builds of in-house developed software, MBR is expected to host these artifacts.

Delivery pipeline for the average software development team
Delivery pipeline for the average software development team

A software project of almost any size must store its binary artifacts in a repository so that they can be accessed by programmers or tools that provide continuous integration (Continuous Integration), delivery (Continuous Delivery) and deployment (Continuous Deployment). Since the quality of the software that manages the repositories has a significant impact on the work of development teams, ICBM must meet the following requirements:

  • Integrability – There are many build automation tools for each programming language, such as Maven for Java, Npm for JavaScript, or Rake for Ruby. There are also many continuous integration tools like Jenkins, CircleCI or Travis. Since automation is the most important aspect of DevOps, plug-ins are expected to be installed for each or most of them to provide seamless integration with MBRs.

  • Availability – Artifacts are essential components of the software development process and they should be available to the programmer or build tool immediately when needed, without even minor delays, and delivered as quickly as possible.

  • Scalability – most build artifacts are large binaries; some of them can be larger than 1 GB, such as Docker images or .war (complete web application for Java). The MBR must be able to support large datasets, up to almost no size restrictions.

  • Reliability – the possibility of data corruption due to software or hardware failures should be excluded as much as possible.

Artipie

Artipie is a free artifact management tool. It’s not just a service, it’s a constructor that includes many components to provide support for almost all widely used package formats, such as: Maven, Docker, Debian, NPM, go, Helm, ruby, Python, Anaconda, HexPm, Composer. Artipie allows you to use various data storage options, such as: file storage, S3– compatible cloud storage, Redis store, and also provides an interface for using a user-implemented store. In addition, there is a functionality that allows you to manage user access and when setting up using organizations – user groups.

Artipie is an open source project that started in 2020 and is currently under active development. All of its components are based on the principles of reactivity, asynchronous communication, and non-blocking threads.

The Artipie architecture consists of 3 main parts:

  1. Artipie HTTP Engine

  2. Repository Adapters

  3. Vaults

Artipie structure
Artipie structure

Artipie HTTP Engine

The Artipie HTTP Engine is a Java application that provides an API for repository access and management operations. It routes requests to the repository adapter and provides authentication and authorization mechanisms. Artipie HTTP engine built on frameworks Vert.x and RxJavawhich provides ease, flexibility and scalability of the project through asynchronous, reactive and non-blocking operations.

The most common workflow for a data load operation in Artipie is as follows: the client sends some binary artifact to the server, which finds the responsible repository adapter to process the request; the repository adapter stores the data in the repository; when completed, it updates the repository’s metadata (some repositories work differently, e.g. Docker uses the metadata as a path).

Artifact loading scheme for Maven repository with S3 implementation for data storage
Artifact loading scheme for Maven repository with S3 implementation for data storage

Repository Adapters

Repository adapters are independent projects aimed at implementing an API for interacting with the repository client and meta-information generation layers for a specific package type (npm, maven, hexpm, etc.). The Artipie HTTP engine uses adapters to provide MBR functionality. Each repository adapter encapsulates an API for accessing binary data and metadata files.

Artipie supports the following types of repositories:

  • Docker – private registry for Docker images.

  • Maven – a repository for Java, Kotlin, Groovy, Scala and Clojure artifacts and dependencies of various types, such as: .jar, .war, .klib, etc.

  • NPM – a repository for accessing JavaScript language code and packages.

  • pypi – directory of Python packages.

  • Anaconda – data science packages for Python, R, Lua, C, C++, etc.RPM – .rpm package repository for RHEL, PCLinuxOS, Fedora, AlmaLinux, CentOS, openSUSE, OpenMandriva, Oracle Linux, etc.

  • Gem – hosting RubyGem for the Ruby language.

  • go – a package repository for the Go language.

  • Storage of binary files – place any files you need.

  • Helm – repository of Helm-charts.

  • NuGet – storage service for .NET packages.

  • Debian – package repository for Debian-based Linux distributions: Ubuntu, MX Linux, Mint, Raspberry Pi OS, Parrot OS, etc.

  • HexPM is a package manager for the Erlang and Elixir ecosystem.

  • Composer – package repository for the PHP language.

The number of adapters is constantly growing and you can see future adapters on Artipie road map.

Vaults

Repository adapters put downloaded and built binaries into a repository. Artipie uses abstract storage, which provides an abstraction over the physical storage system, making it easy to implement an interface to almost any storage system. The store performs two basic operations: pushing and retrieving an item from the store, as well as a few additional functions, such as checking for the existence of an item, getting a list of the store’s items, or getting the item’s metadata.

The following storage implementations currently exist:

  1. Storage in the file system.

  2. Protocol based storage S3.

  3. Redis storage.

The abstract store also provides an interface that can be extended and implemented to support any desired type of store.

Using Artipie

There are two options for working with Artipie:

Next, let’s look at a simple example of running Artipie locally and configuring it to store docker images on a Windows machine.

Training

To run Artipie you need install Docker and Docker Composethen you can just clone GitHub repository with exampleplacing it along the path C:\or follow these steps:

  1. Create a directory for the Artipie project (for example C:\artipie).

  2. Create a directory for the Artipie configuration file (for example C:\artipie\config) and place the Artipie server configuration yml file there. In this example, the file is named artipie.yml. You must specify the path where Artipie will find all the repository configurations. In this example, the path to the docker repository configuration is specified as /var/artipie/repobecause the directory is mounted from the local Windows machine C:\artipie\repo to catalog /var/artipie/repo in docker container Artipie. This is described in docker-compose.yml at step #4.

meta:
  storage:
    type: fs
    path: /var/artipie/repo #path to repository configurations
  layout: flat
  1. Create a directory for the repository configuration files (for example C:\artipie\repo), then put the yml file with the repository configuration in it. The name of this file will be the name of the repository being created. For example, the file is called my-docker.yml and accordingly the repository will be available using the name my-docker. To configure storage with the FileStorage type, it is enough to specify the path where Artipie will store all the items. The system must have read and write permissions to this directory.to create files in that path. In this example, the directory is used /var/artipie/images.

repo:
  type: docker
  storage:
    type: fs   #тип = FileStorage
    path: /var/artipie/images   #место где будут сохранятся файлы
  1. Create a docker-compose.yml file with the following set of instructions:

version: "3.3"
services:
  artipie:
    image: artipie/artipie:latest
    container_name: artipie
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /C/artipie/repo:/var/artipie/repo # монтируем папку с конфигурацией репозитория
      - /C/artipie/config:/etc/artipie/ # монтируем папку с конфигурацией сервера Artipie

launch

Now open the console, go to the folder with the file docker-compose.yml and start Artipie with the command: docker-compose up. The following console output shows that Artipie started on port 8080 and found the repository configuration file my-docker.yml.

C:\>cd C:/artipie
C:\artipie>docker-compose up
[+] Running 2/2
 - Network artipie_default  Created                              0.7s
 - Container artipie        Created                              0.1s
Attaching to artipie
…
artipie  | [INFO] main com.artipie.VertxMain - Artipie was started on port 8080
artipie  | [INFO] ForkJoinPool.commonPool-worker-1 com.artipie.asto.fs.FileStorage - Found 1 objects by the prefix "" in /var/artipie/repo by /var/artipie/repo: [my-docker.yml]
…

Usage

Push image

To create the image that will be used in this example, we will use an already existing image available on docker hub. In this case, it’s a lightweight image with Alpine Linux.

  1. Download the alpine image with the command: docker pull alpine:

C:\artipie>docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
  1. With the following command, we will create our docker image called myfirstimagewhich is based on the image alpineto host it in our Artipie repository: docker image tag alpine localhost:8080/my-docker/myfirstimage. Let’s check that Docker contains 2 images – alpine and localhost:8080/my-docker/myfirstimage, using the command docker images:

C:\artipie>docker image tag alpine localhost:8080/my-docker/myfirstimage
C:\artipie>docker images
REPOSITORY	                           TAG     IMAGE ID      CREATED      SIZE
localhost:8080/my-docker/myfirstimage  latest  e66264b98777  7 weeks ago  5.53MB
alpine                                 latest  e66264b98777  7 weeks ago  5.53MB
  1. Now we need to log into our own registry as a standard user artipie|artipiesince the Docker registry only supports authorized users: docker login --username artipie localhost:8080:

C:\artipie>docker login --username artipie localhost:8080
Password: artipie
Login Succeeded
  1. Finally, we can push this image into our repository with the command: docker push localhost:8080/my-docker/myfirstimage:

C:\artipie>docker push localhost:8080/my-docker/myfirstimage
Using default tag: latest
The push refers to repository [localhost:8080/my-docker/myfirstimage]
24302eb7d908: Pushed
latest: digest: sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253 size: 528
  1. To check the result, we can connect to the container with the command: docker exec -it artipie bash and go to directory/var/artipie/images/my-docker to find the folder docker with created folders and blobs.

Pull image

Before downloading an image from our repository, remove the existing image from the list of local images with the command: docker image rm localhost:8080/my-docker/myfirstimage. Make sure the image has been deleted and there is only 1 image now(alpine): docker images:

C:\artipie>docker image rm localhost:8080/my-docker/myfirstimage
Untagged: localhost:8080/my-docker/myfirstimage:latest
Untagged: localhost:8080/my-docker/myfirstimage@sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253

C:\artipie>docker images
REPOSITORY	     TAG      IMAGE ID       CREATED         SIZE
alpine           latest   e66264b98777   7 weeks ago     5.53MB

The image can be extracted using the command: docker pull localhost:8080/my-docker/myfirstimage. After executing the command, you can verify that two images are now available again (alpine and localhost:8080/my-docker/myfirstimage): docker images:

C:\artipie>docker pull localhost:8080/my-docker/myfirstimage
Using default tag: latest
latest: Pulling from my-docker/myfirstimage
Digest: sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253
Status: Downloaded newer image for localhost:8080/my-docker/myfirstimage:latest
localhost:8080/my-docker/myfirstimage:latest

C:\artipie>docker images
REPOSITORY	                           TAG     IMAGE ID      CREATED      SIZE
localhost:8080/my-docker/myfirstimage  latest  e66264b98777  7 weeks ago  5.53MB
alpine                                 latest  e66264b98777  7 weeks ago  5.53MB

Conclusion

The binary repository is a key element of any DevOps pipeline, and each project requires a different type of binary repository to store artifacts. Almost every programming language ecosystem has its own type of artifact store. Artipie provides a wide range of adapters for major packaging formats in one product, allowing you to manage all your artifacts from one place. It allows you to add your own binary data storage implementation in addition to file storage, Redis and S3.

Artipie is a good choice for storing binary artifacts because it’s easy to set up, grows and evolves quickly, and supports many existing types of repositories. Being an open source product, it requires no payment to use.

More information about Artipie can be found at GitHub. If you have any questions or suggestions – you can ask them in Telegram group.

To support the development of Artipie, feel free to give the project a star on GitHub.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *