Three layers of Kubernetes in Kubernetes


Hi all. My name is Dmitry Putilin (Kind Cat) Telegram.

From the team FR Solutions : We continue the series of articles about K8S.

In this article, we will talk about the architectural concepts for deploying kubernetes clusters.
Let’s touch on both legacy methods and new practices, such as cluster-api and k8s in k8s.

In the past article talked about how a cluster is assembled without using config generators like kubeadm.

2014 Let there be KUB

in the beginning there was a cube, severe and primitive.

Initially, the first cubes were similar to existing installations, but the main difference was the need to deploy them manually.

All subsequent examples have the same set of components during initial installation – 5 binary elements (api, scheduler, controller, proxy, etcd), after which you need to create certificates, configure configuration files with services, configure sysctl and modprobe, and start the service using the command systemctl start.

The installation process itself looks pretty simple, but at a time when few people knew about k8s, it was a challenge.

2015 Kubespray

At the end of 2015, everyone was trying to write their own automation tool for this routine task in Ansible or Puppet. However, in 2015, a group of like-minded people released the first release of the kubespray tool. It was based on ansible and was so successful that people continue to use it to this day.

The idea was as simple as possible – to translate the legacy process into the form of Ansible code and automate it as much as possible. The project developed rapidly and brought important innovations, including:

  • Switching to static pods

  • Ability to deploy and update addons with a single tool.

  • Ability to renew certificates using ansible-playbook play.yaml.

  • Now you can use a combination of Terraform and Kubespray, which allows you to order managed nodes in the cloud and configure them using a single flow.

2016 Kubeadm

Also, at the same time as Kubespray, the development of a tool called Kubeadm began, which was focused on unification and simplicity, not allowing you to go beyond the inherent limitations, but at the same time providing enough flexibility to deploy the required installation.

In my subjective opinion, this tool became popular only about 3-4 years ago and has just begun to gain its audience.

Some of the features that make Kubeadm attractive to users include:

  • Ease of installing and configuring a Kubernetes cluster

  • A unified approach to installing Kubernetes across OSes and platforms

  • Support for various network plugins

  • Automatic discovery of network interfaces

  • Ability to customize scheduler controllers and data nodes according to user requirements

  • Easily add and remove cluster nodes

  • Using kubernetes compatible networking plugins such as Calico, Weave and Flannel to manage the network settings of the cluster.

These features make Kubeadm a convenient and flexible tool for installing and managing Kubernetes clusters.

The disadvantage of this tool is that to use it, you must first configure the node:

  • Initial sysctl and modprobe configuration.

  • Kubelet service with settings.

  • You need to download the kubeadm and kubelet binaries.

  • Creating a basic configuration file for kubeadm.

2018 Cluster API

The concept of this tool is still gaining momentum, we will tell you more about it.

Cluster-API is a tool for automatically creating, managing and scaling Kubernetes clusters based on cloud provider infrastructure. It was designed to provide a unified and unified approach to managing Kubernetes clusters across different cloud providers.

Cluster-API has been focused on cloud infrastructure from the very beginning, and the first providers were developed for AWS and GCP. However, other providers have been added over time, such as VMware vSphere, OpenStack, and Azure.

The main idea of ​​Cluster-API is to take ideas from Terraform and Kubeadm and rethink them in a new context. The Cluster-API allows you to create and manage clusters using simple K8S primitives. It also provides a unified interface for working with different providers, making it more flexible and easy to use in different scenarios.

In this article the history of creating a cluster using Terraform was considered, which represents the first level of K8S IN K8S. The main idea of ​​the article is that to create a cluster, you need to use several resources through Terraform. Some of these resources are already in the Kubernetes base set, while others require creation through CRD + Operator.

* Please note that I specifically write Operator and not Cluster-API, because this approach can be implemented in any way convenient for you, knowing how to write operators.

Terraform vs Operator

The final resources we are interested in look like this, and now we can represent them as K8S resources:

  • Load Balancer is a resource Service Type LoadBalancer

  • VM is a CRD which will describe:

    • MachineClass (Basic machine template)

    • MechineSet (logically like ReplicaSet)

    • MachineDeployment (Logical as Deployment)

    • Machine (logically like a POD)

  • Cloud INIT can also be represented as a custom resource and modified based on additional arguments, which are usually specified in the base kubeadm configuration file.

Since the Cluster-API requires an already existing cluster to work, we use any method convenient for us to create a primary cluster. (First level)

Next, we add the basic functionality required for a cluster created with approach Cluster-API. (Example is given for Yandex Cloud)
* We won’t be able to show the clean Cluster-API because there is currently no provider for Yandex Cloud, in which we usually develop.

As you can see from the diagram above, there is no difference between resources created through Terraform and resources created by the Cluster API approach. All tools request the same resources and work with the API in the same way. The only difference is the unification of the approach.

We can use the same resources to deploy clusters in any cloud using a single approach, and we only need to add special providers (operators) that will implement the configuration for each cloud.

Key features that the Cluster-API approach provides include:

  1. Automated deployment of Kubernetes clusters using declarative descriptions in the form of YAML files.

  2. Unification of the process of creating clusters on various cloud providers and physical servers using common APIs and universal resources.

  3. The ability to scale and manage Kubernetes clusters using operators that provide additional functionality for working with clusters, such as automatic scaling, backups, monitoring, etc.

  4. Centralized control over clusters and their configuration, which makes it easy to manage multiple clusters in different environments.

  5. The ability to integrate with other tools and services, GitOps and CI / CD pipelines, which provides even greater automation and simplification of managing Kubernetes clusters.

The functionality of the Cluster-API has been greatly enhanced by a number of smart decisions and tricks that have allowed it to come forward in development.

Due to the fact that the operator adds all the created nodes to the legacy cluster as workers, we can use the IAM privileges from Kubernetes. Previously, we had to transfer configuration files and certificates for static pods through Day2 tools such as Ansible and Puppet. We can now specify the ConfigMap and Secret in the pod spec from which to retrieve the configuration. This allows you to install Cert-Manager in Kubernetes and issue certificates for the cluster through it, keeping them up to date.

For those who are worried that the first level cluster may fall and access to secrets and configuration cards will be lost, it is always possible to mount files on the host file system and store them there. This allows you to keep access to this information even if the first-tier Kubernetes cluster ceases to function.

Cons of the solution:

The cluster-API approach is not without its drawbacks. One of the main disadvantages is the cost and binding to dedicated workers. If you only have a few clusters, then chances are you won’t think too much about setup costs. But when you have 600 clusters or more, cost can be a significant factor. Besides,
binding to dedicated workers can be a problem if they fail. If you do not have spare workers, then you risk losing access to your clusters.

2017 Kubernetes in Kubernetes

One of the first successful examples of using the concept of Kubernetes in Kubernetes was presented in late 2016 or early 2017 as part of the Tanzu project from VMware. There were also stands from the CoreOS community.

Many people to this day look at the concept of Kubernetes in Kubernetes with apprehension, because they do not understand how this technology works and how to use it.

On the portal Habr articles on this topic have already been published in 2021, which demonstrate how quickly and easily you can deploy a Kubernetes cluster in just 30 seconds. I will not focus on technical details, I will only point out the essence. If you’re interested, read the article above.

The difference from the Cluster API is minimal, the only difference is that we do not place static pods on the worker via cloud ini, but place the entire load in separate namespaces in the form of deployments. Everything else we deploy as a regular application in Kubernetes.

In the Kubernetes cluster of the second level, we created deployments of Kubernetes components in the subnet and created an address for external access and management of the cluster through an LB type service.

Above is a diagram of the organization of components in the namespace. It is worth clarifying that for etcd deployment, you can use two options: StatefulSet (STS) and Deployment. STS has a significant disadvantage, which is that when a node falls, the replica does not move without a pendle, because it is tied to a specific node. In turn, Deployment has another problem – it flexibly moves to other nodes when a node falls, but does not have the ability to flexibly manage dedicated DNS records and PV bindings to specific pods. Nevertheless, you can use some tricks and implement the STS functionality through Deployments.

To implement StatefulSet functionality through Deployments, we create three separate Deployments with one replica each, then associate them with the corresponding services, which have static names. Next, we create three Persistent Volumes (PV) and associate them with each of the Deployments and finally we set the upgrade strategy to “recreate”.

This configuration showed excellent results in terms of fault tolerance and scalability. With this approach, you can manage the load profile on the nodes through affinity, selectors, etc. and flexibly scale the cluster without being tied to specific nodes. This allows you to more flexibly manage the load and optimize it.
distribution.

Cons of the solution:

Conclusion

The story about the three levels of K8S in K8S is a story about balancing the price tag of speed and fault tolerance.

As with any fault-tolerant solution, many factors must be taken into account, such as the presence of a cluster with several data processing centers, an equal amount of resources for each DC in order to reduce the impact of a DC failure on the operation of the system in
in general, and much more. Therefore, before proceeding with the creation of such a solution, it is necessary to carefully consider all the nuances and take the necessary measures to minimize possible risks.

For the virgins of the K8S environment of the third level, an excellent solution, raised, worked, removed or extinguished.

For core infrastructure clusters, we use tier 2 clusters.

Contacts

Thank you for reading this article to the end, we hope you found it useful. We will be glad if you connect to our telegram channel and ask us questions.

telegram community: https://t.me/fraima_ru
telegram me: https://t.me/Dobry_kot

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *