Hashicorp Vault – gcs + gcpckms

Secrets Management and Why is it so Important to Us?

Hello! My name is Evgeniy, I work as Lead DevOps at EXANTE. In this article, we will analyze the life experience of the high availability setup of Hashicorp Vault with gcp storage backend and auto unseal in k8s.

Some time ago, our infrastructure consisted of thousands of virtual and hardware machines, on which our legacy services were located. Configuration files with already written secrets in open form were distributed to these machines, partly manually, partly with the help of chef.

For a variety of reasons, including speeding up code delivery processes, ensuring continuous delivery and secure storage of secrets, and accelerating the deployment of new applications and environments, the decision was made to change the company's strategy.

We decided to make our product cloud native, and to do this we needed to change the approach to development and infrastructure, refactor our legacy services, start moving towards a microservice architecture, deploy services in cloud k8s, and use managed resources (redis, postgres).

In our reality, everything had to be changed – from applications and infrastructure to methods of distributing configs and secrets. Google was chosen as a cloud provider, and Hashicorp Vault was chosen as a secret storage. At the moment, we have successfully covered most of the way.

Why Hashicorp Vault?

We have several reasons:

1. We needed a ready-made tool that can change the situation with secrets management in an instant without the need for global modifications to the code of our applications. This tool should also be able to integrate with chef.

2. Hashicorp Vault is the most popular tool with the required functionality (secret storage, secret access control, secret injection into k8s pods, integration with chef, gitlab ci).

3. Our devops engineers have practical experience in configuring HA and unseal.

4. Possibility to further integrate Secret injection webhook into our k8s cluster.

Where and in what configuration do we deploy Hashicorp Vault?

We deploy Hashicorp Vault in a k8s cluster on a separate node pool.

In terms of configuration, we use the approach of automatically unsealing the storage using gcpckms when the Hashicorp Vault pods are restarted. We use Google Cloud Storage as the storage backend for storing Hashicorp Vault data, as it allows for easy and cheap clustering of the Hashicorp Vault service.

Why is automatic unseal so important to us?

It's all pretty simple here – at the moment, most services receive secrets in the form of environment variables using Vault Secret injection webhook, which is why we need automatic unseal: without it, we will potentially face the unavailability of Hashicorp Vault services and the inability to pass environment variables to services during any manipulations with pods (restart, new release, etc.)

What do you need to have to set up HA with gcs storage backend and gcpckms auto unseal?

1. Service account for Hashicorp Vault.

2. GCS bucket.

3. Keyring + cryptokey.

We create the required entities using terraform.

More about iam policies for service account and about keyring and cryptokey gcpckms bundle can be read in the official documentation.

How do we deploy Hashicorp Vault in k8s?

In k8s cluster we use GitOps approach with Flux CD tool. Accordingly all new applications are deployed to the cluster via Flux CD repository. We will not go into details of GitOps usage in this article.
We use SOPS to encrypt primary secrets (required for deploying Hashicorp Vault). SOPS is an encrypted file editor that supports YAML, JSON, ENV, INI, and BINARY formats and encrypts using AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. We use age, and the decryption key is in k8s secrets.

We configure Hashicorp Vault as follows:

– HA mode (3 replicas).

– GCS storage backend.

– Auto unseal using gcpckms.

First of all, we need to create a secret named vault-gcs, putting our service account in it. We create this file using the template below and encrypt it using SOPS.

apiVersion: v1
kind: Secret
metadata:
    name: vault-gcs
    namespace: vault
type: Opaque
data:
    Vault_gcs_key.json: <base64-serviceaccount>

For Hashicorp Vault to work correctly in the specified configuration, it is necessary to forward it to the podfile with our service account and specify additional environment variables. Thus, we fill the values.yaml file:

server:
  enabled: true
  extraEnvironmentVars:
    GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/vault-gcs/vault_gcs_key.json
    GOOGLE_PROJECT: your-project
  extraVolumes:
  - name: vault-gcs
    path: /vault/userconfig
    type: secret

Next, we continue filling values.yaml. It is important to correctly configure the HA config to work in multiple replicas and support gcs storage backend + gcpckms unseal. Ultimately, values.yaml looks like this:

server:
  enabled: true
  extraEnvironmentVars:
    GOOGLE_APPLICATION_CREDENTIALS: /vault/userconfig/vault-gcs/vault_gcs_key.json
    GOOGLE_PROJECT: your-project
  extraVolumes:
  - name: vault-gcs
    path: /vault/userconfig
    type: secret
  ha:
    config: |
      ui = true
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "gcs" {
        bucket     = "bucket-name"
        ha_enabled = "true"
      }
      seal "gcpckms" {
        project     = "gcp-project-name"
        region      = "global"
        key_ring    = "your-keyring"
        crypto_key  = "your-cryptokey"
      }
    enabled: true
    replicas: 3

Additionally, our helm chart uses internal ingress settings, but they are quite specific and are not required for general understanding. In addition, we disable the use of vault-agent-injector, these parameters can be configured independently based on the example in the official helm-chart.

Installation into the cluster is done using Flux CD and is fully automated, using the official helm chart from hashicorp.

After successful installation we have three pods vault-0, vault-1, vault-2 in the vault namespace. To successfully start the service, it is enough to execute the command:

kubectl exec -ti vault-0 -- vault operator init

As a result of execution we get a root token and five recovery keys.

Highly recommended save this databecause their the loss is fatal and effectively cuts off access to managing the Hashicorp Vault cluster.

We have benchmarked Hashicorp Vault with different storage backends. We ran it with postgresql, consul, aws s3, gcs. In our tests, gcs was about 10x slower than postgresql.

Since the speed of secrets delivery is not critical for us and the priority is the possibility of convenient auto unseal, fast integration with the storage backend, it was decided to use gcs.

Our k8s has been running in this configuration with Hashicorp Vault for two years now, and we have not encountered any issues using this configuration to date.

Results:

Hashicorp Vault for us is a constantly available, single and secure source of truth with access differentiation to secrets and various authorization methods. It is an easy-to-use and easy-to-configure tool with up-to-date documentation. It allows you to integrate Secret injection webhook, which provides the necessary functionality: create subsequent secrets inside the cluster without SOPS encryption, importing them directly from the storage, integrate almost any other tools and import secrets from the storage into them (ci, k8s pods, chef, etc.), but more on that next time!