Local package repositories

Hello! Today I want to share our thoughts on how you can protect your development from some potential risks in modern conditions. Actually, what do we mean? We are talking about the fact that in large projects there are often single points of failure in the CI / CD processes, it can be either a simple code repository or various pipeline systems for building code and delivering it to production environments. If we are talking about system software, then you can simply stop updating it, forbid it to go “out”, but in the case of external repositories, unpleasant surprises can await us.

What do we insure against

We have the following potential risks on the agenda:

  • there are cases of introducing “malicious” (in various senses) code into public repositories of packages, for example, it has already been noticed in npm, but there may still be precedents, no one is insured, even if fixing package versions, no one guarantees that their content will change on public servers

  • connection with the “outside world” may be broken for one reason or another

What are we cool

We spin all the solutions described below in docker using docker-compose, a little about installing on debian/ubuntu

well, of course, we install docker

apt-get install docker.io

then install docker-compose

apt-get install python3-pip
pip3 install docker-compose

for daemonization use systemd-unit

/etc/systemd/system/docker-compose.service

[Unit] 
Description=Docker-compose

[Service] 
WorkingDirectory=/etc 
Type=simple 
ExecStart=/usr/local/bin/docker-compose up 
ExecStop=/usr/local/bin/docker-compose down 
Restart=always 
RestartSec=5s

[Install] 
WantedBy=multi-user.target

Add it to autoload

systemctl enable docker-compose.service

and also create a working directory that we will mount in future containers for data persistence

mkdir /var/data

let’s say a convention – the IP address of the machine where the docker container is running, let it be:

10.0.0.1

General meaning

The general meaning of all these solutions will be that our builders go to the Internet through a proxy server that will cache packages/modules, so that on subsequent calls to the proxy server, packages/modules will be returned from the cache, thus we can commit versions, and also, if external channels are unavailable, we can continue development offline for a while.

NodeJS/NPM

Here we have used the system Verdaccio. We use the 5.6.0 tag deliberately, you can use the more recent tag if you wish.

/etc/docker-compose.yaml

version: "3.7"

services:
  verdaccio:
    image: verdaccio/verdaccio:5.6.0
    ports:
      - 4873:4873
    volumes:
      - /var/data:/verdaccio/storage

Starting the daemon

systemctl start docker-compose.service

the following should appear in the logs

# docker logs etc_verdaccio_1 -n 100
 warn --- config file  - /verdaccio/conf/config.yaml
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - http://0.0.0.0:4873/ - verdaccio/5.6.0

after that, “frontenders” need to:

создать файл .npmrc в котором указать registry=http://10.0.0.1:4783

in details here

there is also a habr article from Yandex – here it is

here is another article on habré

and further

Python

For everyone’s favorite python, we will use devpi

Here you have to “twist” a little. The fact is that the process is divided into two stages:

Let’s create a Dockerfile to build the container (you can safely copy and execute)

mkdir /root/docker-devpi
cd /root/docker-devpi

cat > Dockerfile <<EOD
FROM python:3.8
COPY docker-entrypoint.sh /docker-entrypoint.sh
RUN pip install devpi-server devpi-web devpi-client && devpi-init && chmod +x /docker-entrypoint.sh
COPY pip.conf /etc/pip.conf
ENTRYPOINT ["/docker-entrypoint.sh"]
EOD

cat > docker-entrypoint.sh <<EOD
#!/bin/sh
export PIP_CONFIG_FILE=/etc/pip.conf # задание конфигурации для pip
echo "[RUN]: Launching devpi-server"
exec devpi-server --restrict-modify root --host 0.0.0.0 --port 3141
echo "[RUN]: Builtin command not provided [devpi]"
echo "[RUN]: $@"
exec "$@"
EOD

cat > pip.conf <<EOD
[global]
index-url = http://localhost:3141/root/pypi/+simple/
[search]
index = http://localhost:3141/root/pypi/
EOD

after creating the Dockerfile, we need to build it

cd /root/docker-devpi
docker build -t devpi:latest .

then create a temporary /etc/docker-compose.yaml

version: "3.7"

services:
  devpi:
    image: devpi:latest
    volumes:
      - /var/data:/root/.devpi-tmp      

launch

docker-compose up -d

look at the logs to make sure everything started

docker logs etc_devpi_1 |head

# docker logs etc_devpi_1 |head
[RUN]: Launching devpi-server
2022-03-25 09:04:16,913 INFO  NOCTX Loading node info from /root/.devpi/server/.nodeinfo
2022-03-25 09:04:16,914 INFO  NOCTX wrote nodeinfo to: /root/.devpi/server/.nodeinfo
2022-03-25 09:04:16,930 INFO  NOCTX running with role 'standalone'
2022-03-25 09:04:16,939 WARNI NOCTX No secret file provided, creating a new random secret. Login tokens issued before are invalid. Use --secretfile option to provide a persistent secret. You can create a proper secret with the devpi-gen-secret command.
2022-03-25 09:04:18,583 INFO  NOCTX Found plugin devpi-web-4.0.8.
2022-03-25 09:04:18,746 INFO  NOCTX Using /root/.devpi/server/.indices for Whoosh index files.
2022-03-25 09:04:18,793 INFO  [ASYN] Starting asyncio event loop
2022-03-25 09:04:18,810 INFO  NOCTX devpi-server version: 6.5.0
2022-03-25 09:04:18,810 INFO  NOCTX serverdir: /root/.devpi/server

go to the container to copy the initial-data to the persistent folder

docker exec -ti etc_devpi_1 bash

apt update
apt install rsync
rsync -av /root/.devpi/ /root/.devpi-tmp/

now you can redeem a temporary container

docker-compose down

fix /etc/docker-compose.yaml

version: "3.7"

services:
  devpi:
    image: devpi:latest
    ports:
      - 3141:3141
    volumes:
      - /var/data:/root/.devpi

now we start already as a demon

systemctl start docker-compose.service

when the server starts, indexing of all existing packages on pypi.org will begin. The process takes 1.5 hours and runs in the background.

Setting for developers, you need to create a file /etc/pip.conf:

[global]
index-url = http://10.0.0.1:3141/root/pypi/+simple/
[search]
index = http://10.0.0.1:3141/root/pypi/

now the pip utility will go to the caching server, which in turn will give either cached data, or will go to the Internet and cache new data

golang

For caching Golang packages, we used the solution Athens

Create /etc/docker-compose.yaml

version: "3.7"

services:
  athens:
    image: gomods/athens
    ports:
      - 3000:3000
    environment:
      - ATHENS_DISK_STORAGE_ROOT=/var/data
      - ATHENS_STORAGE_TYPE=disk
      - ATHENS_GO_BINARY_ENV_VARS=GOPROXY=proxy.golang.org,direct
    volumes:
      - /var/data:/var/data

start the daemon

systemctl start docker-compose.service

look at the logs

docker logs etc_athens_1

INFO[7:30AM]: Exporter not specified. Traces won't be exported
2022-03-29 07:30:19.231447 I | Starting application at port :3000

now let’s test the functionality

export GOPROXY=10.0.0.1
go get github.com/spf13/cobra

in the logs we will see our call to the proxy server

INFO[7:35AM]: exit status 1: go list -m: github.com/spf13@latest: invalid github.com/ import path "github.com/spf13"
        http-method=GET http-path=/github.com/spf13/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=6614c138-083c-416a-9bc2-2e49968d367b version=
INFO[7:35AM]: incoming request  http-method=GET http-path=/github.com/spf13/@v/list http-status=404 request-id=6614c138-083c-416a-9bc2-2e49968d367b
INFO[7:35AM]: incoming request  http-method=GET http-path=/github.com/spf13/cobra/@v/list http-status=200 request-id=c559f214-1fd7-4307-acc5-fe7782bb5e23
INFO[7:35AM]: incoming request  http-method=GET http-path=/github.com/spf13/cobra/@v/v1.4.0.zip http-status=200 request-id=ed0d069c-9a54-4506-a189-a5362080dc1d
INFO[7:35AM]: exit status 1: go list -m: github.com@latest: unrecognized import path "github.com": parse https://github.com/?go-get=1: no go-import meta tags ()
        http-method=GET http-path=/github.com/@v/list kind=Not Found module= operation=download.ListHandler ops=[download.ListHandler pool.List protocol.List vcsLister.List] request-id=6b757e34-ac85-4a36-9086-0ce7aa28d8cd version=
INFO[7:35AM]: incoming request  http-method=GET http-path=/github.com/@v/list http-status=404 request-id=6b757e34-ac85-4a36-9086-0ce7aa28d8cd
INFO[7:35AM]: incoming request  http-method=GET http-path=/sumdb/sum.golang.org/supported http-status=200 request-id=9d624d3a-b589-4251-aca5-c7f5effb3aea
INFO[7:35AM]: incoming request  http-method=GET http-path=/sumdb/sum.golang.org/lookup/github.com/cpuguy83/go-md2man/v2@v2.0.1 http-status=200 request-id=02957958-e5d4-43eb-ace7-8d60ee42fc8f
...
...

thus using GOPROXY=10.0.0.1 we let traffic through a proxy server, it will give cached versions of modules, or download and cache

PHP

In this case, we use the RepMan solution, which requires a little more attention and resources, because the PostgreSQL database is used, and several containers are launched, there is registration and authorization of users, the creation of internal projects with a different set of modules

First, clone the repository into a directory /var/data

git clone https://github.com/repman-io/repman.git /var/data

for this solution, I had to slightly modify the systemd unit (change the working directory and set the PWD environment variable)

[Unit] 
Description=Docker-compose

[Service] 
WorkingDirectory=/var/data 
Environment=PWD=/var/data 
Type=simple 
ExecStart=/usr/local/bin/docker-compose up 
ExecStop=/usr/local/bin/docker-compose down 
Restart=always 
RestartSec=5s

[Install] 
WantedBy=multi-user.target

For comfortable work, here you will have to create a dns-name for the repman web application, let’s say it will be repman.example.com

If you are using bind9, then you need to register names in DNS

$ORIGIN example.com.
repman IN A 10.0.0.1
*.repman CNAME repman

edit file /var/data/.env.docker

APP_HOST=repman.example.com

if there is GitLab CE, then we edit another option

APP_GITLAB_API_URL=https://git.example.com

for debugging, you can fix the option APP_DEBUG=1

to send mail, we also edit the settings, let’s say we have some kind of MTA configured on 10.0.0.10 (Exim4, Postfix, it doesn’t matter)

MAILER_DSN=smtp://10.0.0.10:25?verify_peer=false
MAILER_SENDER=repman@example.com

we are ready to launch, start the daemon

systemctl start docker-compose.service
hidden text

Mar 29 07:51:30 localhost systemd[1]: Started Docker-compose.
Mar 29 07:51:31 localhost docker-compose[964362]: Creating network “data_default” with the default driver
Mar 29 07:51:31 localhost docker-compose[964362]: Creating data_database_1 …
Mar 29 07:51:33 localhost docker-compose[964362]: Creating data_database_1 … done
Mar 29 07:51:33 localhost docker-compose[964362]: Creating data_app_1 …
Mar 29 07:51:35 localhost docker-compose[964362]: Creating data_app_1 … done
Mar 29 07:51:35 localhost docker-compose[964362]: Creating data_cron_1 …
Mar 29 07:51:35 localhost docker-compose[964362]: Creating data_nginx_1 …
Mar 29 07:51:35 localhost docker-compose[964362]: Creating data_consumer_1 …
Mar 29 07:51:37 localhost docker-compose[964362]: Creating data_consumer_1 … done
Mar 29 07:51:38 localhost docker-compose[964362]: Creating data_cron_1 … done
Mar 29 07:51:38 localhost docker-compose[964362]: Creating data_nginx_1 … done
Mar 29 07:51:38 localhost docker-compose[964362]: Attaching to data_app_1, data_consumer_1, data_cron_1, data_nginx_1
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 | [OK] Consuming messages from transports “async”.
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 | // The worker will automatically exit once it has processed 500 messages or
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 | // received a stop signal via the messenger:stop-workers command.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [OK] Already at the latest version
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | (“Buddy\Repman\Migrations\Version20210531095502”)
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [OK] The “async” transport was set up successfully.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 | // Quit the worker with CONTROL-C.
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 | // Re-run the command with a -vv option to see logs about consumed messages.
Mar 29 07:51:38 localhost docker-compose[964362]: consumer_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [OK] The “failed” transport was set up successfully.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | Installing assets as hard copies.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ———————————————
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | Bundle Method / Error
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ———————————————
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ✔ NelmioApiDocBundle copy
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ✔ EWZRecaptchaBundle copy
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ———————————————
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ! [NOTE] Some assets were installed via copy. If you make changes to these
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | ! assets you have to run this command again.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [OK] All assets were successfully installed.
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 |
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [29-Mar-2022 07:51:37] NOTICE: fpm is running, pid 1
Mar 29 07:51:38 localhost docker-compose[964362]: app_1 | [29-Mar-2022 07:51:37] NOTICE: ready to handle connections
Mar 29 07:51:38 localhost docker-compose[964362]: nginx_1 | certificate found
Mar 29 07:51:38 localhost docker-compose[964362]: nginx_1 | Starting nginx

as a result, 5 containers will be launched

  • data_cron_1

  • data_nginx_1

  • data_consumer_1

  • data_app_1

  • data_database_1

interface is available at https://repman.example.com

to start using this system, write one command in compose.json

{
    "repositories": [
        {"type": "composer", "url": "https://repo.repman.example.com"},
        {"packagist": false}
    ]
}

after that we execute

compose update --lock

After that, the libraries that are in compose.lock will look at the url repman

Aptmirror

Some time ago, the public repositories of Elastic-co began to give http / 403, respectively, the ability to connect these repositories to install packages fell off

To reach the repository, we had to send traffic to their repository through an American server

Next, put the package

apt-get install aptmirror

create a file /etc/apt/elastic-co-6x.list

############# config ##################
#
set base_path    /var/data/elastic_co/6.x
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb https://artifacts.elastic.co/packages/6.x/apt stable main
clean https://artifacts.elastic.co/packages/6.x/apt

create the necessary directories

mkdir -p /var/data/elastic_co/6.x/{mirror,skel,var}

start synchronization, wait for completion

apt-mirror /etc/apt/elastic-co-6x.list

it remains for us to give the mirror out with the help of nginx here is his config

server {
        listen 80;
        autoindex on;
        location /elastic-co {
                alias /var/data/elastic_co;
        }
        location /elastic-co/7.x {
                alias /var/data/elastic_co/7.x/mirror/artifacts.elastic.co/packages/7.x/apt;
        }
}

to connect our mirror on typewriters, let’s create a file /etc/apt/sources.list.d/elastic-co.list

deb http://10.0.0.1/elastic-co/7.x stable main

we have to download the gpg-key of the repository

cd /var/data/elastic_co
wget https://artifacts.elastic.co/GPG-KEY-elasticsearch

in the same way, you can download any apt repository, store it on your servers and install packages without having access to the Internet

What to pay attention to

  1. In large projects with mass use, there can be a lot of traffic, it is important to take this into account, do monitoring and monitor the load

  2. All these solutions can require tens (and, in the case of apt-mirror, hundreds) of gigabytes of disk space, so you need to take care of this in advance, ideally you also need monitoring with graphs

Links

I throw off a list of links where you can read more about these solutions.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *