Setting up secret management with Yandex Lockbox, AWS Secret Manager, Vault Secrets and shell-operator

When working with secrets, I want to get two options: to simply and centrally manage secrets in the cluster and at the same time take them out of the cluster for security purposes. In this article, we will take a closer look at the work External Secrets Operator in conjunction with Yandex Lockbox, AWS Secret Manager, Vault by HashiCorpas well as our own solution based on the Open Source utility from Flant shell-operator.

What is External Secrets Operator

External Secrets Operator extends Kubernetes with Custom Resources, which define where secrets are located and how they are synchronized. The controller requests secrets from an external API and creates Kubernetes secrets. If the secret in the external API changes, the controller monitors the state of the cluster and updates the secrets.

Let’s start by installing the External Secrets Operator. To do this, we use the simplest method – Helm

helm repo add external-secrets https://charts.external-secrets.io 
helm install external-secrets \
     external-secrets/external-secrets \
     -n external-secrets \
     --create-namespace

  • external-secrets is the release name under which External Secrets will be installed. The release name can be anything, as long as it remains unique within the same Kubernetes cluster.

  • external-secrets/external-secrets is the name of the chart we want to install from the External Secrets repository.

  • -n external-secrets is a flag that indicates that we want to create a new namespace (namespace) with the name external-secrets to install the release.

  • --create-namespace – This is a flag that indicates the need to create a new namespace (namespace), if it does not already exist. If the namespace already exists, this flag can be omitted.

We check that all the pods have started and the operator is working:

kubectl -n external-secrets get pod

NAME                                                READY   STATUS    RESTARTS   AGE
external-secrets-5b9599dbd4-czhnh                   1/1     Running   0          49s
external-secrets-cert-controller-6d45db4b4d-zxmmd   1/1     Running   0          49s
external-secrets-webhook-5b9c855467-cxbcs           1/1     Running   0          49s

Now everything is ready to connect our secrets.

Working with Yandex Lockbox

Setting up a bunch of Yandex Lockbox and External Secrets Operator is well described in the official instructions from Yandex. But we will still briefly go over it.

To get started, we need to create a service account in Yandex Cloud for Lockbox.

It has two roles:

After that, open a service account and create an authorized key:

Save the key file authorized-key.json – we still need it for configuration – and remember the secret identifier (in our example e6qqssbs94tjpvdb78p9).

To create our first secret, you need to go to Lockbox and click on the “Create Secret” button.

Immediately create a KMS key to encrypt secrets:

And fill in all the secret fields:

Now let’s add permissions for our service account to the secret created in the previous steps. To do this, go to the “Permissions” tab and click the “Assign roles” button:

And also add a role for the kms key

Let’s add the key from the service account to the cluster. To do this, you need to run the following command:

kubectl --namespace secrets-test create secret generic yc-auth --from-file=authorized-key=authorized_key.json

Next, we will need to create either SecretStore or ClusterSecretStore. Their main difference is that ClusterSecretStore will be accessible from any namespace, and SecretStore – no (the namespace in this case will be set at the stage of creating the entity SecretStore – in the example below, the line is responsible for this namespace: secrets-test). Let’s create a manifest secret-store-yc.yaml For SecretStore:

---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-yc
  namespace: secrets-test
spec:
  provider:
    yandexlockbox:
      auth:
        authorizedKeySecretRef:
          name: yc-auth
          key: authorized-key

Now let’s deploy it to our cluster:

kubectl apply -f secret-store-yc.yaml

After that we will create ExternalSecret using a file external-secret-yc.yaml.

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret
  namespace: secrets-test
spec:
  refreshInterval: 10m
  secretStoreRef:
    name: secret-store-yc
    kind: SecretStore
  target:
    name: yc-secret
  data:
  - secretKey: yc-key
    remoteRef:
      key: e6qqssbs94tjpvdb78p9
      property: SOME_KEY

For more fine-tuning or using several keys at once, I recommend that you read with documentation By ExternalSecret.

Deployim ExternalSecret to the cluster using the appropriate YAML file:

kubectl apply -f external-secret-yc.yaml

Let’s check if our solution works: if we did everything correctly and we have all the necessary rights, the system will display ExternalSecret in status SecretSynced:

kubectl -n secrets-test get externalsecrets.external-secrets.io
NAME              STORE             REFRESH INTERVAL   STATUS         READY
external-secret   secret-store-yc   10m                SecretSynced   True

In addition, there will be a secret yc-secret. Here is its meaning:

kubectl -n secrets-test get secrets yc-secret -ojson | jq -r '.data | map_values(@base64d)'

{
  "yc-key": "SOME_VALUE"
}

Working with AWS Secret Manager

AWS is doing great too with documentation. Let’s start setting up.

First, let’s create a user to work with secrets – here we will need to go to IAM → Users → Add users.

Set a name

Adding a policy. You can use ready-made from AWS – SecretsManagerReadWrite, or you can create your own. We will follow the second path and create our own:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "arn:aws:secretsmanager:*:111111111111:secret:*"
        }
    ]
}

We attach the policy to a new user and create this user, and after that we will create an access key:

Choose Otheradd a tag and save your data for authorization:

Now go to AWS Secrets Manager and create your first secret:

Now let’s prepare the manifests for connecting to the cluster:

---
apiVersion: v1
kind: Secret
metadata:
  name: aws-creds
  namespace: secrets-test
type: Opaque
stringData:
  accessKey: "AKIA46CKSTMN32RJDFXY"
  secretAccessKey: "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"

---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-aws
  namespace: secrets-test
spec:
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: accessKey
            name: aws-creds
          secretAccessKeySecretRef:
            key: secretAccessKey
            name: aws-creds
      region: eu-central-1
      service: SecretsManager
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-aws
  namespace: secrets-test
spec:
  refreshInterval: 10m
  secretStoreRef:
    name: secret-store-aws
    kind: SecretStore
  target:
    name: aws-secret
    creationPolicy: Owner
  data:
  - secretKey: aws-key
    remoteRef:
      key: aws-test-secret
      property: SOME_AWS_KEY

The result is a secret aws-secret with our value:

kubectl -n secrets-test get secrets aws-secret -ojson | jq '.data | map_values(@base64d)'
{
  "aws-key": "SOME_AWS_VALUE"
}

Working with HashiCorp Cloud Platform (HCP) Vault Secrets

This platform also has smart documentation.

werf helm install vault hashicorp/vault --namespace secrets-hcp-vault

After installation, we need to initialize our vault. Be sure to save the keys for later printing of the vault.

kubectl -n secrets-hcp-vault exec -it vault-0 -- sh
/ $ vault operator init

Unseal Key 1: AOFvS1ZVHpLwdKOIX/PFX1YFH9w213ayxsvKIGapEMam
Unseal Key 2: 3Z32U3CruHi83LJnrfz8ptd3tJRTe5DPghQxYG34JtDS
Unseal Key 3: 1re6jWEJuemQXGvwByQzYR4TH02nfcDcf3glK8m/gAI4
Unseal Key 4: AEV3pLV3ddFndToUu2X40fduM1B2167zmeFj/1EmMqBo
Unseal Key 5: XRK2XDxo9NQzN8EVXcLpNBppkW12v+Dm9Hzqq+CBlNp8
Initial Root Token: hvs.Y87NbYmHLvvaZVH0oIsPskZW

With the help of three printout keys, we launch the repository and go into it under root:

vault operator unseal <key>

vault login <root_key>

Now we can add our test secret to the repository:

vault secrets enable --path=secret kv

vault kv put secret/test-hcpv hcpv-key=HCPV_VALUE

vault list secret

vault kv get secret/test-hcpv

Let’s create a policy to access our secret and a token for further configuration:

vault policy write my-vault-policy - <<EOF
path "secret/*" {
capabilities = ["create", "read", "update", "patch", "delete", "list"]
}
EOF
vault token create -policy=my-vault-policy
Key                  Value
---                  -----
token                hvs.CAESIKVdmmqykGVg6yiqDMeB8oC-lAUI_veOWxBKe2AoY490Gh4KHGh2cy40aFpVUldkVlU5SmhuSFQ1TWo2RGt3OFI
token_accessor       QSvnCPet91BAUn0ZZpmBeiGe
token_duration       768h
token_renewable      true
token_policies       ["default" "my-vault-policy"]
identity_policies    []
policies             ["default" "my-vault-policy"]

After that, similarly to the previous examples, we will prepare manifests for connecting to the cluster:

---
apiVersion: v1
kind: Secret
metadata:
  name: hcpv-creds
  namespace: secrets-test
type: Opaque
stringData:
  hcpvToken: "hvs.CAESIKVdmmqykGVg6yiqDMeB8oC-lAUI_veOWxBKe2AoY490Gh4KHGh2cy40aFpVUldkVlU5SmhuSFQ1TWo2RGt3OFI"

---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-hcpv
  namespace: secrets-test
spec:
  provider:
    vault:
      server: "http://vault.secrets-hcp-vault.svc.cluster.local:8200"
      path: "secret"
      version: "v1"
      auth:
        tokenSecretRef:
          name: "hcpv-creds"
          key: "hcpvToken"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-hcpv
  namespace: secrets-test
spec:
  refreshInterval: 10m
  secretStoreRef:
    name: secret-store-hcpv
    kind: SecretStore
  target:
    name: hcpv-secret
  data:
  - secretKey: hcpv-key
    remoteRef:
      key: secret/test-hcpv
      property: hcpv-key

And finally, we check our secret hcpv-secret with a given value:

kubectl -n secrets-test get secrets hcpv-secret -ojson | jq '.data | map_values(@base64d)'
{
  "hcpv-secret-key": "HCPV_VALUE"
}

Working with shell operator

Shell operator is our Open Source utility to make it easier to create Kubernetes operators.

But what if we don’t want to use secrets as K8s objects? To do this, we created a shell-operator image with Yandex-cli, in which there were hooks that worked either on a schedule or on triggers. Using the operator, you can implement and create any logic.

In one of the Flant projects, we go to Yandex Cloud using a service account and request a JWT Token, which we change every six hours for security. In addition, the shell-operator itself is subscribed to events: as soon as a secret appears in the cluster with the annotation lockbox=yes operator creates a new secret or replaces it, and then deletes unnecessary secrets. Below is an example hook:

hook::trigger() {
  # Copy secrets to the other namespaces.
  for secret in $(kubectl -n ${WATCHED_NAMESPACE} get secret -l lockbox=yes --no-headers -o custom-columns=":metadata.name");
    do
      ELAPSED=`kubectl::get_expire_seconds ${WATCHED_NAMESPACE} ${secret}-iam`
      if (( $ELAPSED > 21600)); then
        echo "${secret}-iam doesn't require the update and have ttl - ${ELAPSED} seconds"
        exit 0
      fi
      kubectl -n ${WATCHED_NAMESPACE} get secret $secret -o json| jq -r ".data.sa" | base64 -d > /tmp/sa.json
      ${YC_BIN} config set service-account-key /tmp/sa.json
      rm /tmp/sa.json
      PAYLOAD=$(${YC_BIN} iam create-token --format json)
      TOKEN=$(echo $PAYLOAD | jq -r .iam_token)
      EXPIRATION=$(echo $PAYLOAD | jq -r .expires_at)
      EXPIRATION_TIMESTAMP=$(date -d ${EXPIRATION} +%s)
      # copy secret with a necessary data
      common::generate_secret $secret ${WATCHED_NAMESPACE} ${EXPIRATION} ${TOKEN} ${EXPIRATION_TIMESTAMP}| kubectl::replace_or_create
  done

  # Delete secrets with the 'secret-copier: yes' label in namespaces except 'default', which are not exist in the 'default' namespace.
  kubectl -n ${WATCHED_NAMESPACE} get secret -o json | \
    jq -r '([.items[] | select(.metadata.labels."lockbox" == "yes").metadata.name]) as $secrets |
             .items[] | select(.metadata.labels."lockbox.managed" == "yes" and ([.metadata.labels."lockbox.parent"] | inside($secrets) | not)) |
             "\(.metadata.namespace) secret \(.metadata.name)"' | \
    while read -r secret
    do
      kubectl delete -n $secret
    done
}

What is important to understand here:

  • Function common::generate_secret() generates JSON to create a Kubernetes secret.

  • Function kubectl::replace_or_create() performs replacement or secret generation.

  • Function kubectl::get_expire_seconds() Gets the number of seconds left before the secret expires.

function common::generate_secret(){
  echo "{
  \"apiVersion\": \"v1\",
  \"kind\": \"Secret\",
  \"metadata\": {
    \"name\": \"$1-iam\",
    \"namespace\": \"$2\",
    \"annotations\": {
      \"lockbox.expires\": \"$3\",
      \"lockbox.expires_timestamp\": \"$5\"
    },
    \"labels\": {
      \"lockbox.parent\": \"$1\",
      \"lockbox.managed\": \"yes\"
    }
  },
  \"type\": \"Opaque\",
  \"stringData\": {
    \"token\": \"$4\"
  }
}"
}
function kubectl::replace_or_create() {
  object=$(cat)

  if ! kubectl get -f - <<< "$object" >/dev/null 2>/dev/null; then
    kubectl create -f - <<< "$object" >/dev/null
  else
    kubectl replace --force -f - <<< "$object" >/dev/null
  fi
}
function kubectl::get_expire_seconds() {
  NAMESPACE=$1
  SECRET=$2
  if PAYLOAD=$(kubectl -n ${NAMESPACE} get secret ${SECRET} -o custom-columns=":metadata.name"); then
    EXPIRATION_TIME=$(kubectl -n ${NAMESPACE} get secret ${SECRET} -o json | jq -r '.metadata.annotations."lockbox.expires_timestamp"' || echo "0" )
  else
    EXPIRATION_TIME="0"
  fi
  NOW=$(date +%s)
  echo $((${EXPIRATION_TIME} - ${NOW}))
}

Conclusion

As we found out, the External Secrets Operator perfectly solves the problem of securely storing secrets outside the cluster. But the choice of storage in the first place will depend on the cloud in which the infrastructure is located, and on the required functionality.

AWS, through a flexible IAM mechanism, allows you to limit the resources available for the current service account. If you need full control, then you can use HCP Vault. True, it will have to be studied deeply enough before being launched in production.

PS

Read also on our blog:

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *