Managing secrets in ArgoCD using argocd-vault-plugin with Kubernetes auth

Theoretical part

To begin with, it is worth noting that in ArgoCD, by default, there is no way to secretly manage credentials (logins, passwords, tokens, etc). The ArgoCD documentation provides list of possible credentials management solutions. In my opinion, the closest to native is the option of working with secrets through argocd-vault-plugin (AVP) using Kubernetes auth in Vault. The scheme is as follows: argocd-repo-server authorizes in Vault using the Service Account token, in the Secret manifest in < ... > substitutes the required value, and creates a secret.

Practical part

Setting up ArgoCD

Below are examples of the necessary additional settings for installing ArgoCD using the official helm chart.

Setting up argocd-server

Setting up the application config argocd-server, to connect the plugin:

  # -- [General Argo CD configuration]
  # @default -- See [values.yaml]
    configManagementPlugins: |
      - name: argocd-vault-plugin
          command: ["argocd-vault-plugin"]
          args: ["generate", "./"]

Create an application main, which will be app of app, and will create the rest of the Application:

  # -- Deploy ArgoCD Applications within this helm release
  # @default -- `[]` (See [values.yaml])
  ## Ref:
    - name: main
      namespace: argocd
      additionalLabels: {}
      additionalAnnotations: {}
      project: default
        targetRevision: master
        path: apps
            - values.yaml
        server: https://kubernetes.default.svc
        namespace: argocd
          prune: true
          selfHeal: true

Setting up argocd-repo-server

It is possible to include an AVP in several ways, but the simplest and least time-consuming option is using initContainer. The scheme is as follows: at the start of the argocd-repo-server application, initContainer downloads the required plugin. Using volume of type emptyDir, we mount the downloaded plugin in argocd-repo-server.

Setting volume and volumeMounts in argocd-repo-server:

  # -- Additional volumeMounts to the repo server main container
    - name: custom-tools
      mountPath: /usr/local/bin/argocd-vault-plugin
      subPath: argocd-vault-plugin

  # -- Additional volumes to the repo server pod
    - name: custom-tools
      emptyDir: {}

It was previously mentioned that requests to get secrets from Vault will be made by the application argocd-repo-server, so he needs to set up a serviceAccount:

  ## Repo server service account
  ## If create is set to true, make sure to uncomment the name and update the rbac section below
    # -- Create repo server service account
    create: true
    # -- Repo server service account name
    name: argocd-repo-server

Configuring initContainer for plugin download:

  # -- Init containers to add to the repo server pods
    - name: download-tools
      image: alpine:3.8
      command: [ sh, -c ]
        - wget -O argocd-vault-plugin
          chmod +x argocd-vault-plugin && mv argocd-vault-plugin /custom-tools
        - mountPath: /custom-tools
          name: custom-tools

The above settings are collected in one file argrocd-values.yaml, and available on github.

Applying the above settings in ArgoCD

For demonstration purposes, you can use the ready-made argocd-values.yaml configuration, and deploy ArgoCD, for example, to minikube.

Clone the repository with the settings:

git clone;
cd  argocd-vault-plugin-demo/helpers

Applying parameters with helm:

helm upgrade -i argocd argo/argo-cd 
    --create-namespace -n argocd 
    -f argocd-values.yaml 

At this point, ArgoCD setup is complete.

Vault setup

If Kubernetes auth is not yet enabled in Vault, it can be enabled with the following command:

vault auth enable kubernetes

In case Kubernetes auth was just enabled with the command above, you need to configure additional parameters with the following command:

vault write auth/kubernetes/config 
  token_reviewer_jwt="$(cat /var/run/secrets/" 

It is worth paying attention to the parameter issuer=”https://kubernetes.default.svc.cluster.local”, which is required for Vault 1.9.0 and Kubernetes 1.20 and above. It is strongly recommended that you read the official Vault documentation for more details. Kubernetes auth method.

To demonstrate, let’s add an additional KV storage of the 2nd version, and put in it test secret:

vault secrets enable -path=avp -version=2 kv
vault kv put avp/test sample=secret

Create and apply a file with policy for argocd-repo-server:

cat << EOF > /tmp/policy.hcl 
path "avp/data/test" { capabilities = ["read"] } 
vault policy write argocd-repo-server /tmp/policy.hcl

Create auth in Kubernetes role for argocd-repo-server:

vault write auth/kubernetes/role/argocd-repo-server 
  bound_service_account_namespaces=argocd policies=argocd-repo-server

Creation of App of secrets

To follow the GitOps approach, you need to add Application to the main git repository that ArgoCD relies on. The added Application should create a Secret with the parameters from the Vault. Below is Application example with env options for AVP:

kind: Application
  name: app-of-secrets
  namespace: argocd
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
      prune: true
      selfHeal: true
    path: apps/secrets
        - name: AVP_TYPE
          value: vault
        - name: AVP_AUTH_TYPE
          value: k8s
        - name: AVP_K8S_ROLE
          value: argocd-repo-server
        - name: VAULT_ADDR
          value: http://vault.vault:8200
      name: argocd-vault-plugin
    targetRevision: master

Secret manifest for application app-of-secrets. It is worth paying attention to the AVP annotation indicating the path to the secret in Vault. The full list of annotations for the plugin is available here

kind: Secret
apiVersion: v1
  name: example-secret
  namespace: default
  annotations: "avp/data/test"
type: Opaque
  sample-secret: <sample>

At this point, the secret in the default namespace should have a value from Vault. You may need to do a hard refresh in ArgoCD UI, or run the command:

argocd app sync --force app-of-secrets


Setting up ArgoCD comes down to: inserting a binary AVP file, creating an Application with env parameters, and creating a Secret with stringData and an annotation. As bonus small was written head script, which automates the deployment of ArgoCD and Vault in minikube. Criticism, wishes, suggestions are welcome.

I would like to express special gratitude for the preparation of images melpomane

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *