Parent helm chart for projects + werf Sequel
In the previous Prequel article, we looked at deploying applications using the werf tool from the Flant company, the purpose of which was to simplify deployment for developers and essentially relieve devops (so that they would not waste time on deploying applications, at least the devs, since essentially it’s all copy-paste), reduce time to market, in this article I want to share the experience of this solution on real projects in retrospect, which turned out to be inconvenient and what we ultimately came to.
Issues
Problem with the charts
Initially, separate Helm charts were developed for different programming languages - PHP, Go and JavaScript. This quickly became awkward to support. For example, when adding functionality (for example, VPA) to a chart for PHP, the same modification had to be manually transferred to charts for other languages. Maintaining consistency between them proved difficult.
Versioning and incompatibilities
Since the chart supports semantic versioning, and the chart itself is loaded on the fly during deployment (we specified a minor version), in case someone got confused with the versions when adding a feature to the chart and it turned out that the version is not compatible or is generally broken, but tagged as a patch, it may happen that all deployments will fall at launch, we do not control the chart update.
Problems with GitLab CI settings
Developers immediately had problems setting up variables in GitLab
Some didn't have access.
Someone didn't know where to find the necessary variables.
There were frequent cases of confusion with variable values (even among DevOps).
To be honest, even I didn’t always remember which variables needed to be added and where. This caused a lot of unnecessary communications and slowed down the deployment process.
Complexity of CI/CD pipeline
Over time the pattern CI/CD became too big and complex. Developers had to spend considerable time studying it, despite the availability of templates and examples. This led to constant questions and communications between developers and DevOps specialists, which slowed down the work process.
Problems managing charts in Git
The absence of heterminism is a serious mistake. Everything about the application should be under version control. In our case, the Helm charts were outside the Git repositoryeven though they are part of the application. This created a version control gap and reduced control over the infrastructure.
Fixes
Unification of Helm charts
We created unified Helm chartsuitable for all programming languages in use, making it much easier to maintain and extend. Now this chart is used to deploy projects to PHP, Go, JavaScript and Java, Payton. This eliminated duplication and the need to sync changes across charts.
Refusal to automatically pump out charts in CI/CD
Instead of loading the chart automatically during deployment, it is now required manually download and store in the application repository. This provided complete version control of the charts.
However, a new problem has arisen: developers without DevOps must be able to load the chart, and not everyone is ready to learn Helm. Documentation with sequential steps is often not effective enough.We solved this problem using skeleton for applications. For example, for Go we created a skeleton that:
The developer can only write business logic and configure values for deployment. We can apply the same approach to PHP and other languages.
Simplifying CI/CD and getting rid of unnecessary variables
Since we removed automatic chart pumping in CI/CD, unnecessary variables are no longer needed in pipelines. This reduced the burden on developers and reduced the number of errors during deployment.
CI/CD templates to simplify deployment
To save developers from having to delve into the intricacies of CI/CD, we have made deployment templates common templates. When installing the application via skeleton, all the necessary templates are connected automatically. Connection example:
include: - project: 'helm-charts/helm-chart' file: '/templates/.gitlab-ci-laravel.yml'This approach also makes it easy maintain templates in one place. For example, if you need to add linting stage for Gowe only change one template and the changes apply to all projects.
Dependency management via Git
Now all application dependencies, including Helm charts, are located managed by Git. This solved the problem where previously, after cloning a repository, it was possible not to find the folder with charts – which violated the principle of heterminism and complicated the work. Now all the necessary deployment components are available in the application repository.
Bottom line
We significantly reduced deployment time: Previously, deploying a finished or even partially finished application could take from one day to a week depending on the workload of the DevOps team. Now this process takes the developer about 30 minutes. Of course, developers still have minor problems, such as typos, but this is no longer critical.
We also relieved the DevOps team: Now developers independently deploy their applications to dev And staging environment, and deployment in production goes through a review process. This approach has worked well in practice. The most important thing is developers now understand not only how to write code, but also what exactly they deploy and where.
Afterword
Why did I write this post? I just know that many teams follow a similar pathand someone has already asked me questions about the implementation. I wanted share experienceaccumulated from working with real projects and interacting with development and DevOps teams (and yes, I myself developer, not DevOps).
I tried my best to make this post simple and clearwithout going into complex technical details, so that it is accessible to all readers. I hope my experience will be useful for those who are faced with similar tasks in their projects or for those who are already working on similar schemes and feel that something is going wrong.
Thank you very much to the guys from Flant for what you do.
