Managing secrets in ArgoCD using argocd-vault-plugin with Kubernetes auth

Theoretical part

To begin with, it is worth noting that in ArgoCD, by default, there is no way to secretly manage credentials (logins, passwords, tokens, etc). The ArgoCD documentation provides list of possible credentials management solutions. In my opinion, the closest to native is the option of working with secrets through argocd-vault-plugin (AVP) using Kubernetes auth in Vault. The scheme is as follows: argocd-repo-server authorizes in Vault using the Service Account token, in the Secret manifest in < ... > substitutes the required value, and creates a secret.

Practical part

Setting up ArgoCD

Below are examples of the necessary additional settings for installing ArgoCD using the official helm chart.

Setting up argocd-server

Setting up the application config argocd-server, to connect the plugin:

  # -- [General Argo CD configuration]
  # @default -- See [values.yaml]
  config:
    configManagementPlugins: |
      - name: argocd-vault-plugin
        generate:
          command: ["argocd-vault-plugin"]
          args: ["generate", "./"]

Create an application main, which will be app of app, and will create the rest of the Application:

  # -- Deploy ArgoCD Applications within this helm release
  # @default -- `[]` (See [values.yaml])
  ## Ref: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/
  additionalApplications:
    - name: main
      namespace: argocd
      additionalLabels: {}
      additionalAnnotations: {}
      project: default
      source:
        repoURL: https://github.com/dmitrii-dmnk/argocd-vault-plugin-demo.git
        targetRevision: master
        path: apps
        helm:
          valueFiles:
            - values.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Setting up argocd-repo-server

It is possible to include an AVP in several ways, but the simplest and least time-consuming option is using initContainer. The scheme is as follows: at the start of the argocd-repo-server application, initContainer downloads the required plugin. Using volume of type emptyDir, we mount the downloaded plugin in argocd-repo-server.

Setting volume and volumeMounts in argocd-repo-server:

  # -- Additional volumeMounts to the repo server main container
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/argocd-vault-plugin
      subPath: argocd-vault-plugin

  # -- Additional volumes to the repo server pod
  volumes:
    - name: custom-tools
      emptyDir: {}

It was previously mentioned that requests to get secrets from Vault will be made by the application argocd-repo-server, so he needs to set up a serviceAccount:

  ## Repo server service account
  ## If create is set to true, make sure to uncomment the name and update the rbac section below
  serviceAccount:
    # -- Create repo server service account
    create: true
    # -- Repo server service account name
    name: argocd-repo-server

Configuring initContainer for plugin download:

  # -- Init containers to add to the repo server pods
  initContainers:
    - name: download-tools
      image: alpine:3.8
      command: [ sh, -c ]
      args:
        - wget -O argocd-vault-plugin
          https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.6.0/argocd-vault-plugin_1.6.0_linux_amd64 &&
          chmod +x argocd-vault-plugin && mv argocd-vault-plugin /custom-tools
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

The above settings are collected in one file argrocd-values.yaml, and available on github.

Applying the above settings in ArgoCD

For demonstration purposes, you can use the ready-made argocd-values.yaml configuration, and deploy ArgoCD, for example, to minikube.

Clone the repository with the settings:

git clone git@github.com:dmitrii-dmnk/argocd-vault-plugin-demo.git;
cd  argocd-vault-plugin-demo/helpers

Applying parameters with helm:

helm upgrade -i argocd argo/argo-cd 
    --atomic 
    --create-namespace -n argocd 
    -f argocd-values.yaml 
    --version=3.29.5

At this point, ArgoCD setup is complete.

Vault setup

If Kubernetes auth is not yet enabled in Vault, it can be enabled with the following command:

vault auth enable kubernetes

In case Kubernetes auth was just enabled with the command above, you need to configure additional parameters with the following command:

vault write auth/kubernetes/config 
  issuer="https://kubernetes.default.svc.cluster.local" 
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" 
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" 
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

It is worth paying attention to the parameter issuer=”https://kubernetes.default.svc.cluster.local”, which is required for Vault 1.9.0 and Kubernetes 1.20 and above. It is strongly recommended that you read the official Vault documentation for more details. Kubernetes auth method.

To demonstrate, let’s add an additional KV storage of the 2nd version, and put in it test secret:

vault secrets enable -path=avp -version=2 kv
vault kv put avp/test sample=secret

Create and apply a file with policy for argocd-repo-server:

cat << EOF > /tmp/policy.hcl 
path "avp/data/test" { capabilities = ["read"] } 
EOF
vault policy write argocd-repo-server /tmp/policy.hcl

Create auth in Kubernetes role for argocd-repo-server:

vault write auth/kubernetes/role/argocd-repo-server 
	bound_service_account_names=argocd-repo-server 
  bound_service_account_namespaces=argocd policies=argocd-repo-server

Creation of App of secrets

To follow the GitOps approach, you need to add Application to the main git repository that ArgoCD relies on. The added Application should create a Secret with the parameters from the Vault. Below is Application example with env options for AVP:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-secrets
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  source:
    path: apps/secrets
    plugin:
      env:
        - name: AVP_TYPE
          value: vault
        - name: AVP_AUTH_TYPE
          value: k8s
        - name: AVP_K8S_ROLE
          value: argocd-repo-server
        - name: VAULT_ADDR
          value: http://vault.vault:8200
      name: argocd-vault-plugin
    repoURL: https://github.com/dmitrii-dmnk/argocd-vault-plugin-demo.git
    targetRevision: master

Secret manifest for application app-of-secrets. It is worth paying attention to the AVP annotation indicating the path to the secret in Vault. The full list of annotations for the plugin is available here argocd-vault-plugin.readthedocs.io

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  namespace: default
  annotations:
    avp.kubernetes.io/path: "avp/data/test"
type: Opaque
stringData:
  sample-secret: <sample>

At this point, the secret in the default namespace should have a value from Vault. You may need to do a hard refresh in ArgoCD UI, or run the command:

argocd app sync --force app-of-secrets

Outcome

Setting up ArgoCD comes down to: inserting a binary AVP file, creating an Application with env parameters, and creating a Secret with stringData and an annotation. As bonus small was written head script, which automates the deployment of ArgoCD and Vault in minikube. Criticism, wishes, suggestions are welcome.

I would like to express special gratitude for the preparation of images melpomane