How We Move Thousands of Secrets to StarVault in Seconds
It would seem like a simple task: transfer secrets between Vault storages. But in practice, difficulties arise. And there are so many of them that we at Orion soft have developed our own utility for migration – StarVault Shuttle.
In this article, I will tell you why we chose this solution and how our utility works. We will also talk about the pitfalls of migration via backups and discuss other tasks of “moving” to the new Vault, for example, the need to “clean up” the storage.
We already they toldhow we decided to create our own StarVault secret storage. Our partners and customers who are interested in migrating from HashiCorp Vault asked: how to quickly and easily transfer secrets from one Vault to another.
There are three ways to do this:
Carrying everything by hand is long and labor-intensive;
Making a backup and restoring is faster, but there are some nuances;
Take a utility that will do everything itself.
How will we migrate?
I think there is no need to go into detail about the inconvenience of manual migration. It is enough that each element needs to be unloaded and loaded into a new environment.
Migration via backup is also not always convenient. You can make a backup copy of the necessary secrets, but you will have to upload it taking into account the Storage Backend. Secret storage can be built on different technologies: file system, Consul, etcd, object storage. If the Storage Backend is different, you will have to do some magic to unload backups, for example, from the object storage, and place them in Integrated Storage (Raft).
If the migration involves a DBMS, you will need a specialist to perform Backup/Restore at the database level. You can find a separate person who will do it for a table in PostgreSQL. Or you can call a shuttle 😉
What is StarVault Shuttle?
We decided to develop our own ready-made solution for migrating secrets from the original Vault storage. The goal was to make the migration as comfortable as possible for the user. We took the open Vault API, wrote the code in Python and Flask. To interact with Vault, we used the HVAC client.
Shuttle supports two modes of secret migration: online and offline.
Online Migration
Everything is simple here:
Launch the utility;
We enter the settings and access details;
Gaining access to the original Vault.
4. Select secrets for migration;
5. Click the migration button. If the target storage parameters were entered correctly, the secrets will immediately appear in the new Vault. It takes a few seconds to migrate 100 secrets.
Migration occurs automatically.
Offline migration
This mode is useful when the old and new Vault circuits are isolated from each other. In this case, you need to unload secrets and load them on the new system. But how is this different from a regular Backup/Restore?
At first glance, the solution looks like backups. But there is an important difference: you don’t need to think about which backend is used in the new storage. Shuttle gives secrets through the Vault API, and the storage itself places them on its backend, whatever it is.
We select the necessary secrets in the secrets tree;
Click “Export”. The encrypted file is saved (AES-256 algorithm);
We write the file to the flash drive and transfer it to the second circuit;
Again, via StarVault Shuttle, we upload secrets to the new storage.
How to work with Shuttle
Shuttle offers two methods of operation – via the console (CLI) and via the web interface.
Working via console
For many sysadmins, the easiest way is to run the script via the CLI:
# Подключение
python start.py --src-vault <src_vault_url> --src-token <src_token> \
--src-key <src_key> --dst-vault <dst_vault_url> \
--dst-token <dst_token> --dst-key <dst_key>
# Выбор Secret Engines из доступных
Available Secret Engines:
1. cubbyhole/
2. identity/
3. kubernetes/
4. kv_test_v1/
5. kv_test_v2/
6. ldap/
7. sys/
8. transit/
Enter the numbers of the engines you want to export separated by spaces: 4 5 8
# Информирование о процессе миграции
KV secrets created successfully in kv_test_v1/
KV secrets created successfully in kv_test_v2/
Transit key testkey created successfully in transit/
Transit key testkey2 created successfully in transit/
Working via web interface
Using the web interface you can:
Control the migration process at every stage;
Choose which secrets to transfer to the new storage;
Get rid of outdated or unnecessary data.
The more migrations, the more urgent the task of “cleaning up” the storage. After all, Vault often contains secrets that are no longer needed by anyone, outdated information, and other artifacts. They may not be in the way, but why move them to a new Vault?
Shuttle allows you to select only the secrets you want to migrate.
What secrets are supported?
Currently the utility supports the following Secret Engines:
Key-Value (KV): versions 1 and 2, configuration and kv-secrets;
SSH: configuration and roles;
Transit: configuration and keys;
LDAP: configuration, roles and libraries;
Kubernetes: configuration and roles;
PKI Certificates: configuration, roles, certificates and keys;
Databases: configuration, connections and roles.
Some Secret Engines are still being improved:
SSH. Currently, only role configurations are migrated. We are working on migrating certificates;
Transit. For now, only configs are migrated. In the near future, we will add migration of encryption keys themselves.
One utility vs. people with patience
Shuttle simplifies and speeds up the process of migrating secrets between Vaults. You can use any vault that supports the Vault API.
With Shuttle, you don't need to involve a DBMS specialist or a person who knows how to make backups, or someone to manually migrate with a lot of patience and the same amount of time.
Our utility will also help:
Conduct a “cleaning” of the storage area;
Change backend;
Merge Vaults;
Move to a new platform.
And all this with minimal time investment.
What do you think? Do you need such a utility? If so, share in the comments what additional functions you would like to see in it?
