restic + MinIO (S3)

Until recently, I’ve been backing up my devices to an external HDD. This method suited me because I rarely backed up data – once a week. When I thought about daily backups, I realized that running with an external HDD would be impractical. I want to tell you about a convenient (at least for me) solution for daily backups without wires.

restic

To create the backups themselves, I use the program restic. This is a simple CLI program that works on all popular operating systems because it is written in Go. Allows you to make backups locally, to external devices and to cloud services out of the box. It compresses files quite effectively, for example, my ~111GB backup of my home directory turned out to be only 20GB. At the first start, it makes a full snapshot of the specified directory, on subsequent runs it backs up the delta and saves only changes as separate snapshots. A snapshot is the contents of a directory at a particular point in time.

MinIO

Backups need to be stored somewhere, I chose S3 storage for this MinIO. It is quite easy to deploy it on your server, it is easy to understand the web interface and in general it production ready solution, it is successfully used in many large data storage projects.

Installation and setup

You can download restic from github repository or using your OS’s package manager.

First of all, we initialize the repository:

restic -r RESTIC_REPOSITORY init

The repository can be a directory or the address of a cloud service, for example, if you make a backup directly on the same computer to a folder backup-repothen the command will look like this:

restic -r /home/backup-repo init
enter password for new repository:
enter password again:

restic ask you to enter the password for the repository, you cannot lose the password, otherwise access to the data will be lost.

The command to start the backup process is as follows:

restic -r /home/backup-repo --verbose backup /documents

Flag --verbose required for detailed information. With this command, we started the backup of the directory documents to the directory backup-repo.

You can restore data from a backup with the following command:

restic -r /home/backup-repo restore latest --target /restore_folder

latest – this is the most recent snapshot, you can specify the id of the desired snapshot, if required. restic allows you to restore individual files and directories for this after the flag --path you must specify the correct path.

It is most convenient to deploy MinIO in a docker container on your server:

docker run \
   -p 9000:9000 \
   -p 9090:9090 \
   --name minio \
   -v /mnt/my-storage:/data \
   -e "MINIO_ROOT_USER=ROOTNAME" \
   -e "MINIO_ROOT_PASSWORD=CHANGEME123" \
   quay.io/minio/minio server /data --console-address ":9090"

-p forwards ports between the container and the machine on which it runs.
9000 the port of the service itself, 9090 web interface port.
-v specifying the directory in which the data will be stored on the host machine, the service will mirror the data to the directory /data.
-e environment variables needed to access the web interface.
--console-address the address where the web interface will be available.

After installing MinIO, you need to go to the console and create an access key so that restic can connect to the storage.

Now you can make a backup in MinIO:

# задаем переменные окружения с ID и SECRET, созданными ранее в веб-интерфейсе
export AWS_ACCESS_KEY_ID=my_key_id
export AWS_SECRET_ACCESS_KEY=my_key_secret

# создаем новый репозиторий и указываем пароль
restic -r s3:http://localhost:9000/backup-repo init
enter password for new repository:
enter password again:

# запускаем процесс бэкапа
restic backup -r s3:http://localhost:9000/backup-repo --verbose /documents

Instead of http://localhost:9000 will be your address on which MinIO is raised.

In general, this is how backups work with restic and MinIO, but making backups by hand every day will quickly get boring, so this process needs to be automated so that the backup process is completely autonomous.

systemd

I use the sytstemd init system for this, but instead of systemd you can use cron on Linux or the task scheduler on Windows to run a script on a schedule.

First of all, you need to create a file in which there will be environment variables for restic to work ~/.config/restic-backup.conf

# ID и SECRET MinIO
AWS_ACCESS_KEY_ID=my_key_id
AWS_SECRET_ACCESS_KEY=my_key_secret
# адрес репозитория MinIO
RESTIC_REPOSITORY=s3:http://localhost:9000/backup-repo
# пароль от репозитория
RESTIC_PASSWORD=restic_pass
# путь к директории, которую бэкапим
BACKUP_PATH="/documents"
# за сколько дней нужно хранить бэкапы
RETENTION_DAYS=7

And the systemd service file itself ~/.config/systemd/user/restic-backup.service

[Unit]
Description=Restic backup service
[Service]
Type=oneshot
ExecStart=restic backup --verbose $BACKUP_PATH
ExecStartPost=restic forget --verbose --keep-daily $RETENTION_DAYS
EnvironmentFile=%h/.config/restic-backup.conf

Command forget needed to delete old snapshots, flag --keep-daily allows you to configure the policy for storing snapshots i.e. for the last n days that have one or more snapshots, keep only the most recent one for each day. forget deletes only snapshots, but not the data itself.

To delete data, we will make another service. The question may arise, why not add prune to the first service, you can do that, but the fact is that the commands will be run at different intervals, so they are divided into two services.

Second service ~/.config/systemd/user/restic-prune.service

[Unit]
Description=Restic backup service (data pruning)
[Service]
Type=oneshot
ExecStart=restic prune
EnvironmentFile=%h/.config/restic-backup.conf

Team prune will clear data referenced by remote snapshots.

Now we will make timers for services in the same directory.

~/.config/systemd/user/restic-backup.timer

[Unit]
Description=Backup with restic daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

OnCalendar sets the start time, my service starts daily at 12 am, so the value is daily.

~/.config/systemd/user/restic-prune.timer

[Unit]
Description=Prune data from the restic repository monthly
[Timer]
# This will run on the 1st of every month at 2AM
OnCalendar=*-*-01 02:00:00
Persistent=true
[Install]
WantedBy=timers.target

The cleaning service will run once a month, every first day at 2 am. I specified 2 am so that there would be no conflict between services accessing the repository.

For the services to work, you need to restart the systemd manager so that they are picked up:

systemctl --user daemon-reload

And start the timers:

systemctl --user enable --now restic-backup.timer
systemctl --user enable --now restic-prune.timer

At this stage, all settings are completed and if everything is done correctly restic will make daily backups and clear the old ones once a month. Instead of MinIO, you can use any other storage, you can see the list supported by restic’om Here.