Guide to rolling out single and multiple applications
Hello everyone! This is Yuri Shakhov, DevOps engineer at Flant. Recently, I needed to organize a seamless deployment of client applications. I studied various approaches for this and settled on the blue-green deployment strategy. But the problem was that I could not find materials with practical examples; the articles I found described only theoretical aspects. So I had to study the blue-green deployment approach myself. And now I wanted to share this experience.
In this article, I will deploy the application to blue-green and also show how the switch between blue and green works in practice. I will not consider various deployment strategies and their advantages. To get acquainted with the theory of blue-green and other variations, I recommend reading our material on various deployment strategies in Kubernetes.
I have divided this article into two parts: first, we will consider the implementation of deploying an application in the blue-green strategy, and then we will try werf bundle for deploying several applications from one repository. There are different ways to implement this strategy, you can use additional tools such as Service Mesh, Argo CD and others. I will deploy using werfdescribe all resources as Helm templates, and use GitLab for deployment. It is assumed that the reader is familiar with these technologies. The peculiarity here is that native entities and mechanisms in the form of labels for Kubernetes are used. Further, green and blue will be called “versions” of the application. Also, in this article, we will not consider issues of database migration, although for some applications this may be necessary.
Simple blue-green
Let's assume that we have an application and we want to deploy it. We will do this in two stages:
Deploy the application itself (
deploy_app), such as Deployment and Service.Change version: deploy Ingress with the desired Service name (
deploy_ingress). At this stage, traffic will be switched between application versions.
Let's represent this as a pipeline:
To implement this, we define a variable deploy_version (we will take the value from GitLab CI, which we will consider later), which will be equal to blue or green and will be substituted into Helm templates. For Deployment and Service we add labels:
{{ $deploy_version := "" }}
{{ if .Values.werf.deploy_version }}
{{ $deploy_version = print "-" .Values.werf.deploy_version }}
{{ end }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Chart.Name }}{{ $deploy_version }}
labels:
app: {{ .Chart.Name }}{{ $deploy_version }}
...
---
apiVersion: v1
kind: Service
metadata:
name: {{ .Chart.Name }}{{ $deploy_version }}
spec:
selector:
app: {{ .Chart.Name }}{{ $deploy_version }}
...In order for traffic to reach the pod, we need to create an Ingress. To access a specific version of the application, we will specify a Service with the desired name (blue or green). In this case, the Ingress template will look like this:
{{ $deploy_version := "" }}
{{ if .Values.werf.deploy_version }}
{{ $deploy_version = print "-" .Values.werf.deploy_version }}
{{ end }}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
labels:
deploy-version: {{ .Values.werf.deploy_version | quote }}
spec:
ingressClassName: nginx
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Chart.Name }}{{ $deploy_version }}
port:
name: http
tls:
- hosts:
- example.com
secretName: {{ .Chart.Name }}-tlsInstead of changing the Ingress, you can change the Service to direct traffic to blue or green Deployment by labels, but it is not recommended to do this. In this case, we will lose the ability to access the Deployment by the Service name in the cluster. This can be a problem for testing the correctness of the update, since it will not be possible to access the new version before all traffic is directed to it.
You can also create a second Ingress leading to an inactive version with a different domain for testing. In this case, you need to close it behind authorization to restrict access.
Now let's look at the pipeline. When deploying an application, you need to declare a variable deploy_version — the version to which the deployment will be performed. For werf, this can be done like this:
werf converge --set "werf.deploy_version=${DEPLOY_VERSION}"Also during deployment we will check that the version is not active, that is, there is no traffic to it, and our rollout will not affect users. The check itself is implemented as follows: we receive information about which Service the Ingress running in the cluster points to, and find there blue or green.
Below is an example of a complete gitlab-ci.yml:
stages:
- deploy_app
- deploy_ingress
.check_upstreams: &check_upstreams
- APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress example --output=custom-columns="SVCs:..service.name" --no-headers --ignore-not-found | awk -F '-' {'print $NF'})
.deploy_app:
stage: deploy_app
script:
- *check_upstreams
- if [[ ${KUBE_CURRENT_ACTIVE} == ${UPSTREAM} ]];
then
tput setaf 9 && echo "Обнаружена попытка деплоя на активную версию, деплой будет остановлен!" && exit 1;
else
werf converge \
--release example-${UPSTREAM} \
--set "werf.deploy_version=${UPSTREAM}";
fi;
allow_failure: false
.deploy_ingress:
stage: converge_ingresses
script:
- *check_upstreams
- if [ ${APP_CURRENT_ACTIVE} == ${DEPLOY_VERSION} ];
then
tput setaf 9 && echo "Обнаружена попытка переключения на активную версию, деплой будет остановлен!" && exit 1;
else
werf converge
--set "werf.deploy_version=${DEPLOY_VERSION}"
fi;
Deploy to blue:
extends: .deploy_app
environment:
name: production
variables:
UPSTREAM: "blue"
Deploy to green:
extends: .deploy_app
environment:
name: production
variables:
UPSTREAM: "green"
Switch to blue:
extends: .deploy_ingress
environment:
name: production
variables:
DEPLOY_VERSION: "blue"
Switch to green:
extends: .deploy_ingress
environment:
name: production
variables:
DEPLOY_VERSION: "green"What we ended up doing:
Adjusted Helm templates for Deployment, Service and Ingress, adding the “color” of our version to them.
We wrote a CI that:
deploys the application to
blueAndgreen;deploy Ingress, which switches traffic to the desired version;
checks that deployments do not occur on the active version.
Now let's move on to the part with bundles.
Deploying multiple applications with werf bundle
Why might bundles be needed? Let's say a client needs to deploy several applications together, and it is more convenient to do this from one repository. The bundle mechanism allows you to publish an application chart and deploy it later without access to a specific Git repository. All that is required is access to the container registry where the bundle is stored. This approach simplifies the process of delivering an application chart.
We will pack applications using werf bundle. We will not dwell on the description of the tool in detail, you can read about its advantages and use cases in the documentation.
Bundle creation occurs in the main application repository, here we will focus only on deployment. In the CI file, we will specify the names of the applications and the corresponding variables for each of them: repository, bundle tag and Ingress name:
variables:
FIRST_REPO_BUNDLE: registry.gitlab.awesome.ru/frontend/first
FIRST_TAG: "0.1"
FIRST_INGRESS: first
...
# apps_for_matrix & apps_for_bash должны содержать одинаковые значения!
.apps_for_matrix: &apps_for_matrix
["FIRST", "SECOND", "THIRD", "FOURTH", "FIFTH"]
.apps_for_bash: &apps_for_bash
APPLICATIONS=("FIRST", "SECOND", "THIRD", "FOURTH", "FIFTH")Compared to the first part, there will already be three stages in the pipeline. Since we are deploying several applications from one repository, we need to make sure that they are all in the same state. To do this, we will implement a version status check job, we will call it check_upstream:
At this stage the following conditions must be met:
The active version is the same for all applications.
The application has no active versions (assuming it has not yet been deployed to the cluster).
stages:
- check_upstreams
- deploy_apps
- deploy_ingresses
.base_werf: &base_werf
- set -x
- type trdl && source $(trdl use werf 2)
- werf version
- type werf && source $(werf ci-env gitlab --verbose --as-file)
.check_upstreams: &check_upstreams
- *base_werf
- *apps_for_bash
- |
GREEN=false
BLUE=false
EMPTY=0
for APP in ${APPLICATIONS[@]}
do
REPOSITORY_INGRESS=${APP}_INGRESS
APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress ${!REPOSITORY_INGRESS} --output=custom-columns="SVCs:..service.name" --no-headers --ignore-not-found | awk -F '-' {'print $NF'})
EMPTY=$((EMPTY+1))
if [[ ${APP_CURRENT_ACTIVE} == "green" ]];
then GREEN=true;
elif [[ ${APP_CURRENT_ACTIVE} == "blue" ]];
then BLUE=true;
elif [[ -z ${APP_CURRENT_ACTIVE} ]];
then EMPTY=$((EMPTY-1));
else
tput setaf 9 && echo "Что-то пошло не так! Статус версий некорректен" && exit 1;
fi;
done
if [[ ${GREEN} != ${BLUE} ]];
then
if [[ ${GREEN} ]]
COLOR="green"
then tput setaf 14 && echo "Статус версий для приложений одинаков — green, можно продолжать деплой";
elif [[ ${BLUE} ]]
COLOR="blue"
then tput setaf 14 && echo "Статус версий для приложений одинаков — blue, можно продолжать деплой";
fi;
elif [[ ${EMPTY} = 0 ]]
then tput setaf 14 && echo "Ingress для данных приложений в кластере не обнаружено, можно продолжать деплой";
else
tput setaf 9 && echo "Статус версий для приложений отличается, деплой будет остановлен!!!" && exit 1;
fi;
Check_upstreams:
stage: check_upstreams
script:
- *check_upstreams
environment:
name: production
when: always
allow_failure: false
The application will be deployed using a bundle. We pass all the necessary arguments to this command and do not forget to specify a different release name (flag --release) for different applications, otherwise the deployment of one will overwrite the deployment of the previous one. Then, using parallel:matrix At the deployment stage, the required number of deployment jobs will be automatically created based on the number of applications:
Below is an example of implementing application deployment in CI:
.deploy_apps: &deploy_apps
stage: deploy_apps
before_script:
- *base_werf
- REPOSITORY_BUNBLE=${REPOSITORY_NAME}_REPO_BUNDLE
- REPOSITORY_TAG=${REPOSITORY_NAME}_TAG
- REPOSITORY_INGRESS=${REPOSITORY_NAME}_INGRESS
- APP_CURRENT_ACTIVE=$(werf kubectl -n ${WERF_NAMESPACE} get ingress ${!REPOSITORY_INGRESS} --output=custom-columns="SVCs:..service.name" --no-headers --ignore-not-found | awk -F '-' {'print $NF'})
- |
if [[ ${APP_CURRENT_ACTIVE} = ${DEPLOY_VERSION} ]];
then tput setaf 9 && echo "Обнаружена попытка деплоя на активную версию, деплой будет остановлен!!!" && exit 1;
fi;
script:
- werf cr login -u nobody -p ${BUNDLE_PULLER_PASSWORD} ${!REPOSITORY_BUNBLE}
- werf bundle apply
--release $(echo ${!REPOSITORY_BUNBLE} | cut -d / -f4)-${DEPLOY_VERSION}-${CI_ENVIRONMENT_SLUG}
--repo ${!REPOSITORY_BUNBLE}
--tag ${!REPOSITORY_TAG}
--set "werf.deploy_version=${DEPLOY_VERSION}"
when: manual
Deploy to Green:
extends: .deploy_apps
stage: deploy_apps
environment:
name: production
parallel:
matrix:
- REPOSITORY_NAME: *apps_for_matrix
variables:
DEPLOY_VERSION: "green"This is how we got a pipeline that allows us to deploy different applications from previously published bundles from one repository.
Conclusion
Blue-green helps you roll out updates to your applications reliably and quickly. This strategy simplifies the process and allows you to test a new version before its full launch. And bundles are especially useful for deploying several applications at once. This makes management and updating more visual and centralized, which is especially important for large projects.
In this article, we looked at deploying applications in a blue-green strategy using GitLab-CI and modified our CI to deploy multiple applications from a single repository. This guide helps you write CI and deploy your application from GitLab. I hope you find it useful.
P.S.
Read also in our blog:
