The Path to GitOps or How We Transferred a Kubernetes Cluster to Argo CD Management
Introduction
If you work with Kubernetes, you most likely use kubectl, kustomize or Helm to deploy services in the cluster. I already wrote an article about the last utility — you can see it here. Then I talked about my experience of implementing this tool for my own workloads and compared the approaches kubectl apply and helm install.
Configuration management in Kubernetes can be done using various tools. In addition to Helm, you can use simple YAML manifests or kustomize. Each of these tools has its own command.
In one git repository you can store:
yaml manifests for kubectl;
kustomization.yaml, yaml manifests and patches for kustomize;
values.yaml for helm.
This approach is called GitOps. It means that all configuration is stored declaratively in a single repository. However, there are some drawbacks: you need to manually create and update manifests. If the cluster is managed by more than one employee, it is important to ensure that all developers agree on changes and commit them to the git repository. In this case, we cannot provide the concept of a single source of truth (SSOT), which the GitOps approach requires.
Table of contents
Hidden text
A bit of theory about Argo CD
Argo CD is a continuous software delivery tool for Kubernetes. Argo CD takes care of all the tasks of synchronizing the Git repository and the Kubernetes cluster. It tracks all changes in the code and then automatically updates the resources in the cluster.
Argo CD is implemented as a Kubernetes controller that constantly monitors running applications and compares the current state (live state) with the target state (desired state). It has a wonderful UI with which you can manage the synchronization process, view the difference between states, and monitor application resources.
Argo CD adds custom resources (CRD) to the cluster, which can be used to describe its configuration. We can interact with Argo CD using a console utility or through a graphical interface. In this article, the second method will be used.
Installation
Let's install Argo CD into a separate namespace:
# kubectl create namespace argocd
namespace/argocd created
# kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
<...>Let's check that all pods have switched to the Running status:
# kubectl -n argocd get pods
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 66s
argocd-applicationset-controller-744b76d7fd-nfl66 1/1 Running 0 67s
argocd-dex-server-5bf5dbc64d-tp9ms 1/1 Running 0 67s
argocd-notifications-controller-84f5bf6896-h48pk 1/1 Running 0 67s
argocd-redis-74b8999f94-m6vsj 1/1 Running 0 67s
argocd-repo-server-57f4899557-bnz46 1/1 Running 0 66s
argocd-server-7bc7b97977-8wdxx 1/1 Running 0 66sWe should also have an argocd-server service, with which we can access the Argo CD API or UI. By default, it type: ClusterIPbut if necessary (I don't recommend it) you can change it to LoadBalancer or NodePortIn this article I will be opening access via kubectl port-forward:
# kubectl -n argocd port-forward svc/argocd-server 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080Now let's go to http://localhost:8080:
Let's get the password for the admin user from Kubernetes Secret:
# kubectl -n argocd get secret/argocd-initial-admin-secret -o json | jq .data.password -r | base64 -d
IKFWGsjONnt5hLV1We'll successfully log in and see that everything is empty:
Connecting the repository
Let's create a secret with information about connecting to the GitHub repository. Since the repository is public, we only need a link:
apiVersion: v1
kind: Secret
metadata:
name: github-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
type: git
url: https://github.com/AzamatKomaev/argo-demo-habr# kubectl apply -f https://raw.githubusercontent.com/AzamatKomaev/argo-demo-habr/main/repo.yaml
secret/github-repo сreatedLet's make sure that the repositories have been successfully connected:
Deploying nginx
Let's start with something simple: deploy three replicas with Nginx with the ClusterIP service. We currently have the following repository structure:
In the apps directory we will store all our applications. Each subdirectory will have app.yaml, which contains the Application resource. In manifests there will be the YAML manifests we are used to.
Before we create app.yaml, let's take a look at its contents:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: nginx
namespace: argocd # тот же самый, где установлен ArgoCD
spec:
project: default # проект по-умолчанию
destination:
server: "https://kubernetes.default.svc" # Kubernetes API адрес. Т.к ArgoCD запущен в тот же кластере, то путь до ClusterIP
namespace: nginx-demo # пространство имен, где будут созданы ресурсы
sources:
- repoURL: https://github.com/AzamatKomaev/argo-demo-habr.git # ссылка на Git-репозиторий
targetRevision: HEAD # указание на ветку, с котрой стоит синхронизировать состояние репозитория
path: apps/nginx/manifests # абсолютный путь до директории с манифестами
syncPolicy:
automated:
prune: true # разрешает удаление ресурса
selfHeal: true # разрешает ArgoCD самому приводить состояние кластера в соответствии с Git-репозиторием
syncOptions:
- CreateNamespace=true # создавать пространство имён, если оно не существуетLet's apply the manifest:
# kubectl apply -f https://raw.githubusercontent.com/AzamatKomaev/argo-demo-habr/main/apps/nginx/app.yaml
application.argoproj.io/nginx createdNow let's look at the Argo CD UI. The nginx application should have appeared there:
Here we can see all deployed Kubernetes resources and their status. Sync OK means that the application resources are synchronized with the Git repository. Healthy shows that all resources are deployed successfully. Let's make sure that all the described resources are in the namespace:
# kubectl -n nginx-demo get all
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-576c6b7b6-227dc 1/1 Running 0 8m33s
pod/nginx-deployment-576c6b7b6-27p4r 1/1 Running 0 8m33s
pod/nginx-deployment-576c6b7b6-gl24h 1/1 Running 0 8m33s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-service ClusterIP 10.43.25.85 <none> 80/TCP 8m33s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 8m33s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-576c6b7b6 3 3 3 8m33sExpanding the Helm chart
Let's now deploy the Helm chart kube-prometheus-stack. With its help, we can deploy all the necessary components for cluster monitoring: kube-state-metrics for generating metrics about the state of the Kubernetes cluster, Prometheus for collecting metrics, and Grafana for visualizing the collected data.
Let's create a monitoring directory inside apps. A little deeper, let's create a directory with the name of the Helm chart and place the app.yaml file there with the following contents:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prometheus
namespace: argocd
spec:
project: default
destination:
server: "https://kubernetes.default.svc"
namespace: monitoring
source:
chart: kube-prometheus-stack
repoURL: https://prometheus-community.github.io/helm-charts
targetRevision: 60.1.0
helm:
releaseName: prometheus
values: |
grafana:
enabled: true
service:
type: NodePort
nodePort: 31234
persistence:
enabled: true
accessModes:
- ReadWriteOnce
size: 5Gi
finalizers:
- kubernetes.io/pvc-protection
defaultRules:
create: false
alertmanager:
enabled: false
prometheus:
enabled: true
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
Now our repository tree looks like this:
This time we need to specify the chart name and version, release name and current values (values.yaml). Pay attention to the last element in the list syncOptions. If the chart contains CRD, you may get an error related to the large size of the resource data. To avoid this error, you need to add the parameter ServerSideApply=true. More about this here.
It is also important to note that Argo CD does not use helm install to install the chart. Instead, it accepts manifests generated by the command helm templateThus, Argo CD takes care of the entire life cycle of the application.
Let's apply the manifest:
# kubectl apply -f https://raw.githubusercontent.com/AzamatKomaev/argo-demo-habr/main/apps/monitoring/kube-prometheus-stack/app.yaml
application.argoproj.io/prometheus createdWe have a second application in Argo CD:
Let's wait until the application state changes to Healthy. In the release values for access to Grafana, we specified service: NodePort And nodePort: 31234.
I use a service with the NodePort type for quick access to Grafana. Don't neglect the security of your applications!
Let's try to go to node_address:31234. Everything works!
App-of-apps pattern
Now we have only two applications. But the cluster can contain 10, 100, 500, 10000 applications…. And in this case we will need to manually accept manifests from Application. There is a way out – App-of-apps.
The idea is that we have a root application that takes control of others. With this scheme, we can make Argo CD itself create and remove applications added to the repositories.
Let's describe such an Application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-app
namespace: argocd
spec:
project: default
destination:
server: "https://kubernetes.default.svc"
namespace: argocd
sources:
- repoURL: https://github.com/AzamatKomaev/argo-demo-habr.git
targetRevision: HEAD
path: apps/
directory:
recurse: true
include: '**/app.yaml'
syncPolicy:
automated:
prune: true
selfHeal: true# kubectl apply -f https://raw.githubusercontent.com/AzamatKomaev/argo-demo-habr/main/root-app.yaml
application.argoproj.io/root-app createdPay attention to the elements directory. recurse: true specifies that Application should search for manifests recursively throughout the apps/ directory. Using include: '**/app.yaml' we tell the application to accept files named app.yaml only. This way, only other, “child” applications will be managed by the “parent” application, and regular YAML manifests will be managed by the latter.
You may think that the above scheme is quite complicated: you need to describe your own app.yaml for each manifest pack, then specify the destination, source(s) and other parameters. Initially, I did it this way: Helm charts were separate applications, and regular manifests were under the control of the root-app. After increasing the number of such resources in the root-app, I decided to split the manifests into Applications, which I think is more correct.
Let's go back to the Argo CD interface. A third application has appeared. Let's go to it and see that it now controls the other two:
We transfer already created services under Argo CD management
I decided to implement Argo CD in our cluster when several dozen applications were deployed in it. I was afraid that there would be problems when moving from the imperative approach to the declarative one that Argo CD offered. There were also concerns that Argo CD would somehow “harm” the already deployed infrastructure. But everything worked out.
I've had cnpg-operator deployed in the cnpg-system namespace for a week now, and a cluster of three replicas in the default namespace:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-db
namespace: default
spec:
bootstrap:
initdb:
database: db
owner: db
secret:
name: db-creds
instances: 3
monitoring:
enablePodMonitor: true
storage:
size: 1Gi
storageClass: local-path
First, let's describe the Application for the operator (apps/cnpg-operator/app.yaml):
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cnpg-operator
namespace: argocd
spec:
project: default
source:
chart: cloudnative-pg
repoURL: https://cloudnative-pg.github.io/charts
targetRevision: 0.22.0
helm:
releaseName: cnpg
destination:
server: "https://kubernetes.default.svc"
namespace: cnpg-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=trueIt is important that the chart version, release name and namespace match what we already have deployed in the cluster. We will not create the application manually, since we have already configured the App-of-apps pattern. We will simply push the changes to the remote repository, wait a bit and see that Argo CD will pull all the changes itself:
Let's go to the cnpg-operator application itself and make sure that the resources remain intact:
The status of the Healthy application. Also note the date of creation of the resources: 7 days .
Let's look at the installed charts:
# helm -n cnpg-system ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cnpg cnpg-system 1 2024-09-12 15:37:54.390475578 +0300 MSK deployed cloudnative-pg-0.22.0 1.24.0As mentioned earlier, Argo CD does not use the helm utility when creating an Application. To stop the chart from being managed by Helm, you need to remove secrets with the type helm.sh/release.v1:
# kubectl -n cnpg-system get secret --field-selector type=helm.sh/release.v1
NAME TYPE DATA AGE
sh.helm.release.v1.cnpg.v1 helm.sh/release.v1 1 7d6h
# kubectl -n cnpg-system delete secret/sh.helm.release.v1.cnpg.v1
secret "sh.helm.release.v1.cnpg.v1" deleted
# helm -n cnpg-system ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIONWe've sorted out the operator, now let's describe the application for the cnpg cluster (apps/cnpg-operator/app.yaml):
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cnpg-cluster
namespace: argocd
spec:
project: default
destination:
server: "https://kubernetes.default.svc"
namespace: default
sources:
- repoURL: https://github.com/AzamatKomaev/argo-demo-habr.git
path: apps/cnpg-cluster/manifests
targetRevision: HEAD
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=trueIn the apps/cnpg-cluster/manifests directory, create cluster.yaml and place the specification of the previously described Cluster there. We will get the following structure:
Let's push the changes to the repository again and make sure Argo CD has pulled in the resources:
Making changes to Application
Previously, I enabled podMonitor for PostgreSQL. This is a resource that specifies how Prometheus should discover and monitor pods. In order for Prometheus to discover them, you need to make the following changes: changes in values.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prometheus
namespace: argocd
spec:
<...>
source:
chart: kube-prometheus-stack
<...>
helm:
releaseName: prometheus
values: |
<...>
prometheus:
enabled: true
prometheusSpec:
<...>
podMonitorSelectorNilUsesHelmValues: false # +
serviceMonitorSelectorNilUsesHelmValues: false # +
Let's go to Grafana and import the dashboard for cnpg:
Lots of screenshots
Argo CD Image Updater
Great, we figured out how to migrate more static services to Argo CD management. The same Prometheus or CNPG cluster is unlikely to be updated every day, unlike its own applications.
Everywhere I have had experience with CI/CD, new versions of applications were rolled out using the Push model: first, an image was built and uploaded to the registry. Then, the image tag (build number or COMMIT_SHA) was taken and the image was updated in the Deployment specification using kubectl apply or helm upgrade.
If you want to transfer your loads under Argo CD control, then you will need Argo CD Image Updater – a tool for automatic image updates. It automatically checks for new images in the registry that are used in Kubernetes and updates them according to the latest version. This is a Pull approach.
I don't use this tool, choosing to stick with the classic approach for CD of my own loads.
End
GitOps is awesome!
