Kubernetes Becomes Vendor-Agnostic After Changing 1.5 Million Lines of Code
In July 2024, Kubernetes version 1.31 was released., which finally eliminated the built-in integrations of cloud providers.
Since Kubernetes v1.7, the Kubernetes project has had the ambitious goal of removing built-in cloud provider integrations ( KEP-2395 ). While these integrations played an important role in the early development and growth of Kubernetes, their removal was driven by two key factors: the growing complexity of maintaining native support for each cloud provider across millions of lines of Go code, and the desire to make Kubernetes a truly vendor-neutral platform.
After many releases, all cloud provider integrations were successfully migrated from the main Kubernetes repository to external plugins. In addition to achieving the initial goals, Kubernetes was significantly optimized, removing about 1.5 million lines of code and reducing the binary sizes of core components by about 40%.
This migration was complex and time-consuming due to the numerous components involved and critical code paths that relied on built-in integrations for the five initial cloud providers: Google Cloud, AWS, Azure, OpenStack, and vSphere. To successfully complete this migration, four new subsystems had to be built from scratch:
Cloud Controller Manager ( KEP-2392 )
Network Proxy API ( KEP-1281 )
kubelet Credential Provider Plugins ( KEP-2133 )
Each subsystem was critical to achieving full feature parity with native capabilities and required multiple releases to bring each subsystem to a maturity level with a safe and secure migration path. More details on each subsystem below.
Cloud Controller Manager
The Cloud Controller Manager was the first external component introduced in this work. It was required to replace the functionality in kube-controller-manager and kubelet that directly interacted with cloud APIs. This important component is responsible for provisioning nodes by applying metadata tags that indicate the cloud region and zone in which the node operates, as well as IP addresses known only to the cloud provider. It also runs the Service Controller, which is responsible for provisioning cloud load balancers for LoadBalancer-type services.
To learn more, you can check out Cloud Controller Manager section in the Kubernetes documentation.
API server network proxy
The API Server Network Proxy project, started in 2018 in collaboration with the API Machinery SIG, aimed to replace the SSH tunneling functionality in kube-apiserver. This tunneling was used to securely proxy traffic between the Kubernetes control plane and nodes, but it relied heavily on provider implementation details built into kube-apiserver to establish these SSH tunnels.
API Server Network Proxy is now a GA-level extension point in kube-apiserver. It offers a general-purpose proxying mechanism that can route traffic from an API server to nodes through a secure proxy, eliminating the need for the API server to know about the specific cloud provider it is running on. This project also introduced Project Konnectivity, which is increasingly being adopted in production environments.
More information about the Network Proxy API can be found in the file README .
Credential provider plugins for kubelet
The Kubelet Credential Provider plugin was designed to replace kubelet's built-in functionality for dynamically retrieving credentials for image registries hosted in Google Cloud, AWS, or Azure. The legacy capability was convenient because it allowed kubelet to seamlessly retrieve short-lived tokens for pulling images from GCR, ECR, or ACR. However, as with other areas of Kubernetes, supporting this required kubelet to have some knowledge of different cloud environments and APIs.
Introduced in 2019, the credential provider plugin mechanism provides a common extension point for kubelet to execute plugin binaries that dynamically provide credentials for images hosted on different clouds. This extensibility extends kubelet's ability to retrieve short-lived tokens beyond the original three cloud providers.
To learn more, you can read the article “Kubelet Credential Provider for Authenticated Image Pulls“.
Migrating storage plugin from in-tree to CSI
The Container Storage Interface (CSI) is a control plane standard for managing block and file storage systems in Kubernetes and other container orchestrators that became generally available in version 1.13. It was designed to replace the in-tree volume plugins built directly into Kubernetes with drivers that can run as a Pod in a Kubernetes cluster. These drivers communicate with kube-controller-manager storage controllers via the Kubernetes API and with kubelet via a local gRPC endpoint. There are now over 100 CSI drivers available for all major cloud and storage providers, making stateful workloads on Kubernetes a reality.
However, there was a major problem in how to deal with all the existing in-tree volume APIs. To maintain API backward compatibility, an API translation layer was built into the controllers that translates the in-tree volume API to the equivalent CSI API. This allowed all storage operations to be redirected to the CSI driver, opening the door for us to remove the code for the in-tree volume plugins without removing the API.
For more information on In-tree Storage migration, see the article Kubernetes In-Tree to CSI Volume Migration Moves to Beta.
What's next?
This migration has been a major focus for the Cloud Provider SIG over the past few years. With this major milestone reached, the community’s efforts will be focused on exploring new and innovative ways to better integrate Kubernetes with cloud providers, leveraging the external subsystems that have been built over the years. This includes increasing the intelligence of Kubernetes in hybrid environments, where nodes in a cluster can run on both public and private clouds, as well as providing better tools and frameworks for external provider developers to simplify and optimize their integration efforts.
With all the new features, tools, and frameworks planned, the Cloud Provider SIG isn’t forgetting about the other side of the equation: testing. Another area that the SIG will focus on in the future is improving cloud controller testing to include more providers. The ultimate goal of these efforts is to create a testing framework that includes as many providers as possible so that we can give the Kubernetes community the highest level of confidence in their Kubernetes environments.
Starting with v1.31, cloud providers will be permanently disabled and removed from the Kubernetes core components.
If you have multiple microservices and don't want to spend time setting up CI/CD, monitoring and operating Kubernetes, try our cloud – Amvera Cloud. Amvera – is an alternative to managed Kubernetes. With us, you can update your services via simple commits in Git and get almost all the benefits of Kubernetes without thinking about administration, CI/CD setup, monitoring, alerting and related services. It will be much cheaper than classic managed k8s. Kubernetes is inside us, but you are completely abstracted from its administration, you just need to link your Git repository to the service and update applications with the “git push amvera master” command.
And if you have a complex project and you need help in building infrastructure based on Kubernetes, or just a consultation, write to me at telegram (Kirill Kosolapov), or leave your contact information on page our DevOps team. We don't charge for asking, if it's a small consultation or something simple, we'll help for free. And if it's complicated, we'll negotiate.
