Testing Kubernetes Clusters with Sonobuoy

Today, Kubernetes dominates the market as the primary container orchestration tool, making testing these systems extremely important. For most standard tests, it is sufficient to use the command kubectlhowever, for more in-depth testing of Kubernetes version certification, the Sonobuoy tool is ideal.

Sonobuoy is a Kubernetes cluster diagnostic and testing tool used to check the cluster's compliance with CNCF (Cloud Native Computing Foundation) standards and specifications and to detect potential problems in the cluster's operation.

Sonobuoy runs a series of tests on the Kubernetes cluster, including tests from the Kubernetes End-to-End (E2E) test suite, to verify that the cluster is configured and running correctly. These tests include checks for compliance with CNCF standards and may also include additional plugins for advanced testing.

Sonobuoy provides documentationwhich describes in detail how to work with the tool, I will only briefly tell you about the main steps: about installation, launch, removal and about receiving a report on the test results.

Who will find this article useful?

The article will be useful for those who are involved in/interested in testing Kubernetes clusters, have heard about the Sonobuoy tool, but have never launched it. This article is a quick start that will help you install Sonobuoy and run your first tests.

Background

I work as a tester in a team that develops a product. Managed Kubernetes. The product allows you to deploy and work with fault-tolerant and autoscaling Kubernetes clusters. To work with the cluster, we provide clients with Managed Kubernetes API (not to be confused with Kubernetes API), which I am testing. Of course, API testing alone won't get you far and you need to test the cluster “from the inside” and, by the way, I also write such tests, but I couldn't ignore such a powerful tool as Sonobuoy.

Installation

Download from github latest release, or use the commands:

# Windows

wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.56.0/sonobuoy_0.56.0_windows_amd64.tar.gz
tar.exe -xvf sonobuoy_0.56.0_windows_amd64.tar.gz

# Linux

wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.56.0/sonobuoy_0.56.0_linux_amd64.tar.gz tar -xvf sonobuoy_0.56.0_linux_amd64.tar.gz

Running tests

To check that everything has been installed correctly, run

sonobuoy run --wait --mode quick

This command will run a quick test, if everything is set up correctly, the output should look something like this:

11:44:40          PLUGIN        NODE    STATUS   RESULT   PROGRESS
11:44:40             e2e      global   running                    
11:44:40    systemd-logs      node-1   running                    
11:44:40    systemd-logs      node-2   running                    
11:44:40 
11:44:40 Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.
11:45:00             e2e      global   complete   passed   1/1 (0 failures)
11:45:00    systemd-logs      node-1   complete   passed                   
11:45:00    systemd-logs      node-2   complete   passed                   
11:45:00 Sonobuoy plugins have completed. Preparing results for download.
11:45:20 Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

Key --wait means that we are waiting for the tests to complete, and the progress of the tests will be displayed in the terminal. If you do not specify the key, you can monitor the progress with the command

sonobuoy status

To view the logs, use the command

sonobuoy logs

Key --mode quick means we are running quick tests. Available mods to run can be obtained with the command

sonobuoy modes

To run all tests, you can use the command

sonobuoy run

To run a specific test case, you can use the key --e2e-focusand to skip certain test cases --e2e-skipexample

sonobuoy run run --e2e-focus AdmissionWebHook

Receiving a report

The report can be obtained by running the command

sonobuoy retrieve

Removal

After completing the tests, remove the tool from the cluster using the command

sonobuoy delete

Summary

As I wrote above, this is an article for a quick start, someday I’ll gain strength (and skill too 🙂 ) and write a more detailed article about Sonobuoy.

By the way, I run Sonobuoy tests together with my tests, I wrote a small wrapper client in Python, with the help of this client I run individual test cases and process the report for further loading into Allur. Maybe I'll write about this someday too.