Vanilla All the Way. Vanilla Cloud Solution on K8s

We are publishing a translation of an article about Vanilla Stack – a new cloud-based open-source technology based on Kubernetes.

I recently came across a technology stack Vanilla stackwhich includes many open source components. In this article, we will take a quick look at the installation process and show you the different use cases.

Vanilla Stack at a glance

Vanilla stack can be defined as a cluster Kubernetes with a lot of open-source components.

Among them:

• Rook – manages distributed storage (file system, block, object),
• Openstack – provides infrastructure as a service (IaaS),
• Cloud foundry – Provides Platform as a Service (PaaS).

The following diagram from the official documentation illustrates the global stack architecture.

Note: All solutions that come with Vanilla Stack (Rook, Cloud Foundry, OpenStack and other tools) work as pods in the underlying Kubernetes cluster.

As we can see from the diagram above, users can:

• run applications in the basic Kubernetes cluster through the creation of deployments, services, Helm charts, etc.
• use Cloud Foundry to run the application from code (with a simple cf push);
• use OpenStack to create new infrastructures (virtual machines, software-defined networking, etc.).

Note: The hardware requirements depend on the components that will be installed in the stack.

The following diagram is more detailed, it contains a list of all the components, organized by category, that can be supplied with the Vanilla Stack:

I think this diagram is pretty accurate. Some components are installed by default, others can be selected during installation.

In this article, we’ll walk through the main steps in the installation process. To simplify the task, we will install Vanilla Stack on six virtual machines prepared by the hoster: three will act as the master Kubernetes nodes, and the rest will act as workers.

Note: Cloud providers should be selected depending on the components that we want to install, as some of them may not meet the installation requirements. For example, to install Rook, we need an infrastructure provider that provides block storage.

We start the installation

There are two ways to install Vanilla Stack, they are described in download section… We will be using a Docker installation which can be started with the following command:

$ docker run 
  --name VanillaStack-installer -d 
  -p 8080:8080 
  harbor.vanillastack.io/vanillastack/installer:latest

The installer’s web interface is accessible through port 8080. The installation can be completed in ten easy steps, which we will discuss next.

Requirements

First step, Start, defines the hardware and software requirements for the machines on which the stack will be installed. In our case, we will use six virtual machines. Each one runs Ubuntu 20.04 and has 4GB RAM / 2 CPUs.

As we wrote earlier, three virtual machines will act as the main Kubernetes nodes, and the rest will be working.

# IP addresses of the master nodes
master1 - 159.65.92.123
master2 - 178.62.32.212
master3 - 178.62.40.225

# IP addresses of the worker nodes
worker1 - 178.62.41.87
worker2 - 178.62.27.97
worker3 - 178.62.67.29

In addition, we need to set up a domain (and a couple of subdomains) to access the cluster. We will come back to this step later when configuring the load balancer.

Terms and Conditions

Acceptance of terms of use (Terms) Is our second step. Be sure to read the specified license information.

General settings

Next comes the step General Settings… It is necessary to indicate the type of installation that suits us: the number of master and worker nodes and the initial load.

In our case, we will choose a HA installation with three master nodes, the three remaining nodes acting as workers. Rook is selected by default for repository installation. We are not using OpenStack or Cloud Foundry here. This could be a topic for another article.

Access to the virtual machine by SSH key

Section Public Key provides an RSA public key, it must be copied to each of the virtual machines. This step is required to allow the installation machine (which is running the installer) to run Ansible playbooks to configure each VM.

Note: Ansible Is a configuration management system. It only runs commands on machines over an SSH connection. Ansible does not require an agent to be installed, unlike analogs such as Chef and Puppet…

Instructions for copying the key are indicated in the installer:

• save the key in a file named key.pub…
• copy the key to the target computer using ssh-copy-id:

ssh-copy-id -f -i key.pub <username>@<ip-address-of-node>

• repeat this with all the nodes.

For Windows, follow the instructions from of this articlestarting with Copy SSH Key to Remote Linux Device.

Define Node Information

At the stage Nodes we need to define the IP addresses and user of each of the master and worker nodes.

We also need to define the nodes that will launch the Rook Pods. According to the requirements, we use all three worker nodes.

Checking node configuration

On a step Node-Check we check if all nodes meet the requirements.

Before clicking on the Validates Nodes button, it is important to scroll all the way down and make sure that we are following the Rook rules: a raw block device must be connected to each node.

Using the command lsblk -f, we can check if everything is configured correctly.

By pressing the button Validate Nodes in the installer, we can verify that all nodes are configured correctly.

Cluster settings

The first thing to set up at the stage Cluster-Settings, Are Pod CIDR and Service CIDR (that is, the IP ranges that are used to provide pod IP addresses and service virtual IP addresses). In our case, we use the default values.

In addition, we need to create a load balancer to redirect traffic to the master or production node.

Due to the nature of the cloud provider you are using, you cannot use the same load balancer to redirect traffic to a different set of nodes.

Therefore, we will create two load balancers: in front of the main nodes
and in front of the workers.

Load balancer in front of main nodes

The first load balancer is required to access the cluster API server. It should be configured like this:

• redirect TCP traffic from port 6443 to the same port on master hosts;
• regularly check the operation of master nodes by sending TCP requests to port 6443.

In this case, the load balancer gets the IP address 159.65.211.35.

Load balancer in front of worker nodes

The second load balancer is required to provide access to the cluster ingress controller (the component that gives access to applications in the cluster). Its settings:

• redirect HTTP traffic (port 80) to port 30080 of worker nodes;
• redirect HTTPS traffic (port 443) to port 30443 of working nodes;
• regularly check the health of worker nodes sending TCP requests on port 30080.

In this case, the load balancer gets IP 46.101.64.165.

Configuring DNS entries

The load balancer IP address must be associated with the cluster domain name. In our article we use vanilla.techwhale.io… Additional subdomains are required to access the API server and applications exposed through the ingress controller.

At this stage, we can only provide the IP address of one load balancer, but in the next releases of the installer there will be more options.

Let’s Encrypt setup

Let’s Encrypt Is a widely used free certificate center. It allows us to receive and automatically renew HTTPS certificates. Kubernetes Let’s Encrypt is often used via Cert Manager, a component installed by default in the Vanilla Stack. Cert Manager can obtain a certificate from Let’s Encrypt (as well as several other sources) and make it available to the application through Kubernetes Secret.

Rook Configuration

The following steps allow you to select components Rookthat will be installed (in this case – the Rook control panel and monitoring), as well as the number of repetitions of one piece of data (replica level).

Installing additional tools

In chapter Additional Tools you can specify which components will be installed in the cluster.

The tools listed here are pretty common. They fall into different categories:

• monitoring (Prometheus, Jeager);
• logging (fluentd, Elasticsearch, Grafana);
• container registry and Helm charts repository (Harbor);
• ingress controller (nginx).

This is a convenient way to install all components from one place. They can also be installed later if they are not selected at this stage.

Subscription

If we want to subscribe to commercial support, then at this step we can get all the necessary information.

Support is provided by the service Cloudical… It supports Vanilla stack and provides Vanilla cloud, a new managed stack.

Checking all steps

Here we can view and change (if necessary) the configuration parameters.

Before moving on to the next step and installing the entire stack, we need to make sure that
/etc/apt/sources.list configured correctly on each node and directed to the official repositories.

Note: This step is necessary, for example, if Ansible is not available from the default repositories.

The easiest way to fix this is to replace the content /etc/apt/sources.list with the following instructions:

Installation

Now everything is ready and we can start the installation under a cup of tea.

In the installation logs, we will see Ansible playbooks applied to each VM in order to:

• install all required packages;
• set up a Kubernetes cluster (three main nodes / three workers);
• install Rook and all the tools selected above.

Installing the entire stack shouldn’t take long.

Note: When installing the stack, I ran into a small problem due to the peculiarities of the cloud provider used: the load balancer redirects traffic to the internal (private) IP address of the host instead of the public one. The Ingress controller is mapped to the external (public) IP address of the nodes, so the worker nodes are marked as bad, which is wrong.

Cluster access

File kubeconfig can be learned from /etc/kubernetes/admin.conf file…

# Get kubeconfig file
$ ssh root@master1 cat /etc/kubernetes/admin.conf > vanilla.cfg
# Configure local kubectl
$ export KUBECONFIG=$PWD/vanilla.cfg

As usual, we start by listing the nodes in our cluster:

$ kubectl get no
NAME      STATUS   ROLES    AGE   VERSION
master1   Ready    master   38m   v1.19.6
master2   Ready    master   37m   v1.19.6
master3   Ready    master   37m   v1.19.6
worker1   Ready    worker   37m   v1.19.6
worker2   Ready    worker   37m   v1.19.6
worker3   Ready    worker   37m   v1.19.6

In addition, we can check all the pods running in the cluster, as well as the services that provide them.

It looks like everything is working! The next step can be a test run, for example, a stateful application, which:

• uses block storage provided by a Ceph cluster created through Rook;
• forwarded through the nginx ingress controller;
• uses a TLS certificate from Let’s Encrypt CA, automatically generated by the Cert Manager.

Outcome

The Vanilla Stack setup was quick and the Docker installer is simple and straightforward. Additional parameters should be added soon for more convenience.

If you don’t want to install Vanilla Stack, you can use the same stack directly from Vanilla cloud… This new cloud offering launched in December 2020 and the first beta is now available.

I recommend taking a closer look at this stack as it is a great way to learn more about popular open source projects.