GitOps – Bad and Ugly

OTUS expert – Vladimir Drozdetskiy invites everyone to free webinar, within which he will tell in detail about the course program “DevOps Practices and Tools” and will answer your questions. And right now, according to an established tradition, we are sharing with you an interesting translation.


I recently spoke with developers from Humanitec (this is a Continuous Delivery platform for Kubernetes). Humanitec is interesting in that, contrary to current trends, it is not based on GitOps.

Personally, I’m a big fan of GitOps because it allows you to build CI / CDs without complex tools using only Git and declarative configuration descriptions. But despite the fact that I recently wrote an article “11 Reasons for Adopting GitOps“(11 Reasons to Implement GitOps), in my practice I have repeatedly encountered the limitations of this approach. A conversation with the guys at Humanitec prompted me to write about this negative experience in order to provide you with a more objective picture of GitOps and talk about alternative approaches.

What’s wrong with GitOps?

Not intended for automatic updates

Applications are frequently updated during continuous delivery, and the delivery of these updates needs to be automated. Upon successful completion of the continuous integration process, the CI process must create a Pull Request to roll out the release into the test environment.

However, editing and resolving conflicts in Git is done manually. And a situation can easily arise when several CI processes end up writing to the same GitOps repository, which leads to a conflict.

The conflict does not arise in separate files, but between the two processes that cloned the repository, when one of them pushes before the other. If after that another process tries to push, then its local copy will be outdated, so it will have to pull, and then push again. At this stage, if the system is loaded enough, conflicts may arise with some other process. The reasons for this all lie directly in how Git works. This effect can be reduced by using more repositories (for example, one repository per namespace).

Solving this problem in one of the projects, by adding a complex retry mechanism to the Jenkins Groovy scripts, cost us significant costs.

Increasing the number of Git repositories

Depending on how you map your GitOps repos to application deployment environments (see the previous section), the number of Git repositories can grow with each new application or environment. You also need to configure the appropriate access rights in these repositories and connect the synchronization agents on different clusters. (A sync agent is a process or pipeline that monitors a GitOps repository and synchronizes its content with the required application environment.)

The team I worked with spent over 30% of all development time in a complex corporate environment automating the provisioning of GitOps repositories. This problem can be mitigated by using fewer repositories, such as one repository per cluster. But this will increase the load on a particular repository in terms of access control and Pull Request management. Most importantly, it will only exacerbate the automatic update problem described in the previous section.

Lack of transparency

GitOps promises transparency about what’s going on in your environment, since all target state is stored in Git as plain text. But this only works for relatively simple configurations with a small number of GitOps repositories containing an observable number of configuration files.

In a corporate environment, the number of GitOps repositories and / or configuration files is growing at an incredible rate. As a result, viewing text files becomes not such a convenient way to find answers to questions. For example, even the question of how often certain applications are deployed is difficult to answer because changes in Git repositories are difficult to map to application deployments. Some changes may result in the deployment of multiple applications, and some are just minor configuration changes.

Doesn’t solve the problem of centralized management of secrets

Complex corporate environments require secret management solutions outside of the usual CI / CD process. You should thoroughly audit secrets such as private keys or passwords to access databases. It makes sense to store them centrally in a secure vault like the Hashicorp Vault.

GitOps doesn’t interfere with this approach, but it doesn’t help much either. Git repositories are not the best place to store secrets, because you have to encrypt and decrypt them and they will forever be remembered in the history of Git. Also, with a large number of repositories, secrets are distributed across them, which makes it difficult to track the places where the secret needs to be updated when it changes.

Audit is not as good as it seems

GitOps repositories are a great tool for auditing processes as they keep a complete history of all changes. Therefore, it is easy to answer the question: “What happened to this environment?”

But since GitOps repositories store versions of text files, other questions become more difficult to answer. For example, answering the question “When did you deploy application X?” Requires a study of Git history and full-text search in text files, which is difficult to implement and is fraught with errors.

Lack of validation of input data

If the Git repository is the interface between the Kubernetes cluster and the CI / CD process, then there is no easy way to check the committed files. Imagine you are making an API call instead of Git PR. In this case, the request is validated, and in the case of GitOps, all verification of the manifest and Helm files falls on the user.

Another solution?

All of these shortcomings do not mean that GitOps is a poor solution for application release and configuration changes. This approach offers many benefits without the need for heavy tools. You just need to be aware of the problems that you will face along the way and take them into account in your projects.

So, is there a solution that has all the benefits of GitOps without the disadvantages listed? Let’s first highlight what we would like to keep:

  • Logging of all environment changes.

  • Description and configuration of environments in a declarative form.

  • Process for approving / agreeing on environmental changes.

  • Control over who can make changes to environments.

  • View the target state of the environment and check it against the real state.

As mentioned earlier, the problem is that although Git has all of these features, it is not designed for frequent automatic updates. Also, Git is not suitable for analyzing the data stored in it. The obvious solution is an API service with its own database, and a similar GitOps agent-based synchronization process. (If you are reading this article on a small screen, you can download the diagram here.)

All previous versions of manifests and Helm diagrams are stored in the database. This way, we can support a large number of API updates and remove the awkward conflict resolution process with Git (unless we run into real conflicts, which are very unlikely in this scenario). The approval process for change requests is implemented using API and database calls. RBAC is implemented in a similar way.

All of this is quite expensive to implement. But here we can get additional functionality:

  • Structured database search (How often is application X deployed?).

  • One centralized system serving all environments: there is no increase in the number of git repositories.

  • Easy configuration management of multiple environments. You can implement a hierarchy of configurations.

  • Centralized management of secrets or integration with third party products.

  • Validation of input data.

Since the above solution is much more expensive to implement than a GitOps-based solution, you will either have to create your own internal implementation or use a third-party one.

The most popular of these tools is Spinnaker… In its turn, Humanitec Is the next generation fully focused on Kubernetes. Some of the above features have already been implemented in it, and some are in the plans. We see great potential in such systems as an alternative to GitOps.

Learn more about the course.


Read more:

  • Minimum Viable Kubernetes

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *