Let's talk about Kubernetes
Hi all! My name is Andrey, I am a DevOps engineer.
When I started studying Kubernetes (K8s), I re-read a lot of articles, and both in the articles and in the documentation, the information was very scattered and fragmentary. It was difficult to put the information together into a coherent whole.
At that time, I would like to find one large article, albeit not completely, but in sufficient detail describing the processes of managing the K8s cluster, deploying and servicing applications in it.
I tried to write something like this article.
This article will not include:
instructions for starting a cluster. These operations are described in detail in many articles by Kubernetes.
basic instructions for working with kubectl, command line, etc.
This article will:
generalization, systematization of knowledge about how Kubernetes works.
A fairly detailed analysis of the configurations required to launch and manage applications running in K8s.
So let's get started. Kubernetes is difficult.
Usually, in articles on Kubernetes, they suggest starting your acquaintance with launching Minkube or, for example, studying the documentation, getting acquainted with K8s on https://labs.play-with-k8s.com/undergo laboratory work there.
This is certainly good, but far from reality. In practice, everything is somewhat more complicated.
I repeat, I will not describe the process of launching a cluster here, but we will have to talk about it, about the K8s architecture, to understand the principles of its operation.
There are many options for launching a cluster, depending on the infrastructure in which it is supposed to be used and the needs that caused such a need.
For example, a number of cloud providers offer managed Kubernetes
Here, for a certain fee, as providers tell us, “in a couple of clicks” a cluster is raised, ready for application deployment, but, as a rule, this is quite an expensive pleasure.
Software development companies usually have their own server infrastructure. This means that for application development purposes, you can raise a local Kubernetes cluster in it.
This option is called Self-Hosted, and there are a lot of solutions and tools for turning around.
For example kubeadm or kubespray
Probably the most popular tool/path to deploy Self – Hosted K8s is kubespray
But there are plenty of options here too. For example, Ingress can run on Nginx or HAProxy, you can regulate the number of master or worker nodes, etc. and so on. In general, all cases should be considered depending on the needs.
As you understand, there can be a lot of solutions for launching a Kubernetes cluster, but the architecture of the cluster itself, the principles of management, access organization, and running applications in it are approximately the same.
Now let's consider the following configuration.
Components and Architecture
Let's scratch the surface of the architecture of Kubernetes itself and its important components.
A Kubernetes cluster consists of worker nodes (Nodes).
The cluster given in the example consists of a node with the Ingress role, a master node and two worker nodes. Outside the cluster there is also an Nginx Reverse Proxy server. It is necessary for the convenience of access control, logging, for security purposes, and acts as a single entry point for clients.
Here I would like to clarify that many products and systems, separated from each other, can be launched in one cluster. Access to them is organized through Ingress, using the “host” header described in it. But more on that later.
When looking at the system architecture, we can break it down into services that run on each node and cluster control plane services. Each Kubernetes node runs the services necessary to manage nodes with a master node and to run applications. Of course, each node runs Docker. It handles downloading images and running containers.
Let's look at what the nodes in the cluster consist of.
Master node (Control plane) manages worker nodes and pods in the cluster. Consists of:
kube-apiserver is a Kubernetes component that is an API server that provides interaction with the Kubernetes cluster.
kube-scheduler – schedules placement of pods on cluster nodes. Binds unrunning pods to nodes via the /binding API call.
etcd is a distributed key-value storage for all cluster data. It is not necessarily located inside the master; it can stand as a separate cluster. The master state is stored in the etcd instance. This ensures reliable storage of configuration data and timely notification of other components about changes in state.
controller-manager – launches controllers, consists of:
Node controller: Responsible for detecting and responding when nodes fail.
Job controller: Keeps track of job objects, which represent one-time tasks, and then creates modules to execute them.
EndpointSlice controller: Responsible for managing EndpointSlice objects. Provides communication between services and modules.
ServiceAccount controller: Responsible for managing ServiceAccount objects. ServiceAccount is a resource in Kubernetes that is used to authenticate and authorize applications and services in the cluster.
kubelet is a Kubernetes component that runs on every node in the cluster and is responsible for managing the lifecycle of pods.
Worker nodes consist of components:
kubelet
kube-proxy – a simple proxy balancer is launched on each node. Responsible for implementing virtual IPs, setting proxy rules and filtering traffic.
A pod is a container, or a group of containers with shared sections, launched as a single unit (shared IP).
Ingress node differs from a worker node in that running application pods is prohibited on it, and the Ingress controller is running on it, in our case, on Nginx.
Kubectl (Command Line Interface) is a command line utility that is used to interact with a Kubernetes cluster. Using kubectl you can manage cluster resources such as pods, services, replications, secrets and much more. The kubectl commands allow you to create, update, delete Kubernetes resources, and obtain information about the state of the cluster.
So, we have discussed the basic architecture of Kubernetes and the principles of its operation. Next, I propose to talk about Kubernetes objects, the configuration of which allows you to host and manage applications in the cluster.
Namespace is the ability to divide a physical Kubernetes cluster into virtual ones, each of which is isolated from the others. An abstract object that logically demarcates and isolates resources between pods. It can be considered as an internal virtual cluster that isolates projects or users from each other. Can apply different quota policies to its projects or issue access rights only to a certain area.
Let's look at the configuration:
apiVersion: v1
kind: Namespace
metadata:
name: test
labels:
name: testkind: NamespaceHere you set the type of the object being described.
metadata:
name: testHere the object is given a name.
labels:
name: testLabels are key/value pairs that are attached to objects such as pods, services, replications, and other resources in the cluster. Used to organize and identify resources.
Each object that we will talk about further can be described in a file in the format .yaml and give it to the kubectl utility. For example, for the Namespace above it would look like this:
kubectl apply –f test_namespace.yaml
Running this command will create a new Namespace if it does not exist. When we configure any such object, we tell the cluster that we want to see it in a certain state. Kubernetes will check the current state against the desired one, and if the current state is different from the desired one, it brings it to the desired one.
Let me give you another example. Let's say we have a pod running with a container created from docker image 1. We need to make sure that this container is launched from image 2.
The image for this container is described in the Deployment object. We'll talk about it in more detail below. In order to launch a container from the desired image, you need to change the current Deployment, add a new image to it and apply the changes via kubectl. Then the cluster will see that the running container does not correspond to the required state, and will bring it to the desired state.
Deployment – a controller that manages the deployment state of pods, which is described in the manifest, monitors the deletion and creation of pod instances. Manages ReplicaSet controllers.
Using Deployment, you can define the desired number of pod replicas, container image, launch parameters, and other parameters.
Let's look at an example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
namespace: test
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
imagePullPolicy: Always
ports:
- containerPort: 80
env:
- name: ENV1
value: "value1"
- name: ENV2
value: "value2"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
volumeMounts:
- name: frontend
mountPath: /etc/nginx/conf.d/test.conf
subPath: test.conf
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Sample
initialDelaySeconds: 3
failureThreshold: 30
periodSeconds: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: frontend
configMap:
name: frontend
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1metadata:
name: test-deployment
namespace: testHere we bind the pod with the container described below to the namespace created earlier.
labels:
app: nginxWe assign a label to the described application so that other entities can refer to it.
spec:
replicas: 1We set the number of pods with a running container (horizontal scaling). This can be done using ReplicaSet, but deployments gives more options.
spec:
containers:
- name: nginx
image: nginx:1.14.2
imagePullPolicy: Always
ports:
- containerPort: 80This describes the specifications of the container that runs in the pod.
env:
- name: ENV1
value: "value1"
- name: ENV2
value: "value2"Here the environment variables are passed to the container. In this case, given as an example.
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"This describes the restrictions on the use of cluster resources. This gives predictability of behavior, in terms of resource use, at times of heavy load.
volumeMounts:
- name: frontend
mountPath: /etc/nginx/conf.d/test.conf
subPath: test.confYou can mount volumes of different types to the container.
livenessProbe:
httpGet:
path: /healthz
port: 80
httpHeaders:
- name: Custom-Header
value: Sample
initialDelaySeconds: 3
failureThreshold: 30
periodSeconds: 3
readinessProbe:
httpGet:
path: /ready
port: 80
initialDelaySeconds: 5
periodSeconds: 5Liveness and Readiness tests are needed to monitor the state of the container and restart it if the tests fail. Here are examples.
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1Describes the strategy for launching new pods when updating the deployment.
Let's take a closer look at some of the parameters:
requests, limits (Requests and resource limits).
Why it’s needed: increasing the stability and performance of both individual applications and the cluster as a whole. Prevent resource contention and ensure that applications are not terminated unexpectedly due to lack of resources.
If the node running the pod has enough resources available, the container can (and is allowed to) use more resources than specified in its requests for that resource. However, a container is not allowed to use more than its resource limit (limits).
Request reserves the amount of resources for a container, limits limits use. memory is specified in bytes (Gi, Mi, Ki), cpu in CPU units (1 core, 1m).
Volumes. You can mount volume to containers running in pods. There are many types of them, they are quite well described in documentation.
In my example I will describe configMap.
Using Nginx as an example, it is convenient to show and use in practice mounting the Nginx configuration file in a container, in the /etc/nginx/conf.d directory.
To do this, you first need to describe a separate configMap object, then describe how to mount it in Deployment, as in the example above.
Example ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: test
name: frontend
data:
test.conf: |
server {
listen 80;
server_name frontend.sample.ru;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
access_log /var/log/nginx/err.log;
error_log /var/log/nginx/err.log;
client_max_body_size 100m;
location / {
root /opt/app;
index index.html;
try_files $uri $uri/ /index.html;
}
}Kubernetes probes – These are checks that are carried out during the life cycle of the pod. These are described for each pod container. There are three types of checks.
Startup probe – runs immediately after the pod starts and is used for applications that have a long initialization procedure. Until it is completed, no other probes are run.
Readiness probe – checks the pod’s readiness to process traffic (the pod is not added to traffic routing in service (more on that below) if this test fails).
Liveness probe – checks whether the application is functioning (if the test is not successful, the process in the pod container is restarted).
Strategy – determines the strategy for replacing old pods with new ones.
Kubernetes has two default strategies: Recreate and RollingUpdate. If the strategy is not explicitly described in Deployment, RollingUpdate will be applied.
RollingUpdate is that before old pods are deleted, new ones are launched. Old pods will be deleted only after the new ones have been successfully launched. Combined with the use of probes, zero application downtime is achieved.
When applying the Recreate strategy, old pods are shut down first, and only then new ones are started.
So, using the examples given above, you can understand how to operate namespaces and run applications in a cluster. Next, we’ll look at what we need to access applications outside the cluster.
Services is an object designed to provide access to an application running as a pod or set of pods in a Kubernetes cluster.
Service performs several important functions:
DNS name resolution: the service provides a symbolic DNS name, formed based on the service name, application namespace and cluster DNS suffix for uniform access to applications. Plus, services with the externalName type can describe arbitrary A-records in the internal DNS of the cluster.
Routing and traffic balancing: the service can provide an IP address, when accessed, it balances traffic to application pods using the round robin algorithm, or it can provide IP addresses of application pods in response to a request to its DNS name, to organize balancing at the application level (headless mode with the clusterIp setting: no).
Service discovery: based on the labels described in it, the service searches for pods that match these labels, have passed the readiness test and adds them to traffic routing.
Publishing applications: We can use services to provide external users with access to applications in the cluster.
Types of services:
In order to indicate where our service will be located, whether it will be available externally or only within the cluster, different types of services are used.
ClusterIP – assigns an IP in the service network, which is only available within the cluster. This is the default value and is used if you do not explicitly specify a service type. You can provide access to a service outside the cluster using Ingress. More on this below.
NodePort – provides access to the application via port/ports on the IP address of each node (node). Allows you to use your own external balancing solutions. At the same time, the internal ClusterIP service is automatically created.
LoadBalancer – used by cloud providers such as Google Cloud. The service will be available through your provider's external load balancer, and NodePorts will be created with ports where traffic from the provider and ClusterIP will arrive.
ClusterIP
NodePort
Load Ballancer
Example of a service description:
apiVersion: v1
kind: Service
metadata:
namespace: test
name: frontend
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: nginx
Since the service type is not specified, a service with the ClusterIP type will be launched. To provide access to an application outside the cluster, Ingress is needed.
If you start a service with the NodePort type, you can specify a specific port on which the application in the service will be available. If a port is not specified, it will be selected randomly from the range of ports from 30000 to 32767.
selector:
app: nginx
Here we indicate which application to give to the service.
Ingress serves to organize access to the application outside the cluster. Acts as an “intelligent router” or entry point into the cluster. By publishing it, we will be able to deliver traffic through it to the application within the cluster, based on routing by domain names, URLs/locations, headers and cookies.
Ingress Controller – processes traffic. Its configuration is formed from all Ingress within one cluster.
Let's look at the Ingress example:
kind: Ingress
metadata:
name: frontend
namespace: test
annotations:
nginx.ingress.kubernetes.io/client-max-body-size: "100m"
nginx.ingress.kubernetes.io/client-body-buffer-size: "100m"
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/proxy-buffer-size: "100m"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header custom-header "my_custom_header";
spec:
ingressClassName: nginx
rules:
- host: product.sample.ru
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: frontend
port:
number: 80
Let's look at the configuration:
annotations:
nginx.ingress.kubernetes.io/client-max-body-size: "100m"
nginx.ingress.kubernetes.io/client-body-buffer-size: "100m"
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
nginx.ingress.kubernetes.io/proxy-buffer-size: "100m"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header custom-header "my_custom_header";
Nginx in Ingress can be configured quite flexibly, but this is the subject of a separate article. Several examples are described here.
rules:
- host: product.sample.ru
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: frontend
port:
number: 80
The rules for processing traffic are described here. First of all, this is the “host” header, the request location (path) and the name of the service to which the request will be redirected.
I mentioned above that in my cluster configuration, there is an external Nginx Reverse Proxy. An example of a simple configuration for organizing access to an application in a cluster:
server {
listen 80;
server_name product.sample.ru;
location / {
client_max_body_size 200M;
proxy_pass http://<ip ingress ноды>:<порт, на котором запущен Ingress> ;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass_request_headers on;
proxy_set_header host $host;
}
}
As a result
In this article, I discussed the basic concepts that are necessary to understand how Kubernetes works, how applications run on it, and what entities are needed for this.
There are a huge number of options for cluster configurations, approaches to managing and placing applications in them. I have described only one of many options, but the basic principles of operation and control remain the same. If you dig deeper, you will no longer get an article, but a full-fledged book. They are in sufficient quantity.
I hope my article will be useful to you. Good luck mastering Kubernetes.
