Managing secrets during deployment in k8s
Hello, habr!
At a certain point, we come to understand that a process that works “well” must start working “correctly”, especially when it comes to application secrets.
Given: gitlab(onprem), cloud (in my case Yandex Cloud), 10+ services that need to be actively and frequently deployed (with the ability to quickly increase the number of services)
Requirements for the solution: deployments must occur without the intervention of an engineer directly during the deployment process, the process is transparent to the developer and meets the company’s quality and safety requirements.
Solution: Let me make a reservation right away – my solution is built on Yandex Cloud, but it is valid for any cloud or onprem solution.
I pass the path from the file deployment locally through kubectl to Helm (possibly in a separate topic)
So there are repositories with code, we place . {environment}.values.yaml
When deploying, specify –values=/path/to/file
The main question arises – if the configmap is stored in the repository (via helm variables – this is correct), then what to do with the secrets?
Option 1: use helm-secrets, sops and other solutions
We receive encrypted files with secrets in the repository. When deploying, we specify the path to the file, helm decrypts and deploys the application along with the secrets. However, overhead immediately comes in the form of a system for transferring the key with which you encrypted, be sure not to forget to add *.dec to gitignore, otherwise at some point the secrets leak into the repository and what was the point then? Well, there is a great risk of losing the key and secrets at the same time. The main disadvantage of the approach is that when you change the secret, you need to go through the entire pipeline again, which means checks, code validation, and tests take a long time.
Option 2: use secret management through external systems.
Additionally we will need:
- A place to keep secrets. I use Yandex Cloud Lockbox, you can use, for example, Vault or similar systems
- External secret operator as a tool for synchronizing secrets
- Kyverno as a tool for executing policies on the k8s cluster
The choice of secret storage falls on the shoulders of the systems engineer or architect. I can tell you from experience – I used vault and lockbox. The first one is, of course, more powerful, but for a small enterprise the second solution is enough.
Next, we deploy External Secret Operator in the cluster
Create a service account with a role
lockbox.editor
export HELM_EXPERIMENTAL_OCI=1 && \
helm pull oci://cr.yandex/yc-marketplace/yandex-cloud/external-secrets/chart/external-secrets \
--version 0.5.5 \
--untar && \
helm install \
--namespace external-secret-operator \
--create-namespace \
--set-file auth.json=sa-key.json \
external-secrets ./external-secrets/
To work with secrets, we will also deploy secretstore. It is worth noting that secretstore works in namespace mode and isolates secrets within one ns, ClusterSecretStore allows you to share secrets between different ns. This should be remembered and not once again fumbling around for secrets that will not be used in two or more ns.
First create a secret (in the example yc-auth) with the keys from the service account created earlier. Then set secretstore to the desired namespace
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
name: secret-store
spec:
provider:
yandexlockbox:
auth:
authorizedKeySecretRef:
name: yc-auth
key: authorized-key'At this stage, the preparatory work has been completed.
You can add a template for external-secret to the application chart (or change the example to use it like a regular manifest)
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
namespace: {{.Values.namespace }}
name: {{ include "helm-chart.name" . }}
spec:
refreshInterval: {{ .Values.secretRefreshInterval }}
secretStoreRef:
name: secret-store
kind: SecretStore
target:
name: {{ include "helm-chart.name" . }}
data:
- secretKey: DATABASE_URL
remoteRef:
key: {{ .Values.secretRemote }}
property: DATABASE_URLIn this example, an external-secret resource is created, which is named after the name of the chart, causing the creation of a k8s secret of the same name from the external secret.
Let's pay attention to two parameters:
secretRefreshInterval — interval with which the secret-operator will poll the lockbox to change the secret (default 1h0m0s)
secretRemote – secret identifier in lockbox (the latest current version of the secret will be captured)
After applying the manifest, make sure the resource is in synced state
At this point, we received managed secrets in k8s, we change them in the lockbox – after some time – they change them in Kubernetes. But this is not enough, the deployment will not know that you have changed the secret or added a new variable to it.
This is where the Kyverno tool comes into the picture (if necessary, I will write a separate article about setting it up). For now, let's assume that you already have it.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deployment-restart-policy
namespace: test-ns
annotations:
policies.kyverno.io/title: Restart Deployment On Secret Change
policies.kyverno.io/category: Other
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Deployment
kyverno.io/kyverno-version: 1.9.0
policies.kyverno.io/minversion: 1.7.0
kyverno.io/kubernetes-version: "1.27"
spec:
schemaValidation: false
rules:
- name: update-secret
match:
any:
- resources:
kinds:
- Secret
names:
- test-database-adapter
namespaces:
- sup
preconditions:
all:
- key: "{{request.operation || 'BACKGROUND'}}"
operator: Equals
value: UPDATE
mutate:
targets:
- apiVersion: apps/v1
kind: Deployment
name: test-database-adapter
namespace: test-ns
patchStrategicMerge:
spec:
template:
metadata:
annotations:
ops.corp.com/triggerrestart: "{{request.object.metadata.resourceVersion}}"
This policy does not claim to be a standard, but it fulfills its task – namely, to restart the deployment if the checksum of the specified secret changes.
As a result, what solution did we get:
When the application is deployed, an external secret is created, which is responsible for updating the k8s secret. Changing the latter serves as a trigger to restart the application.
