Open benchmarks for load testing of servers and web applications

This is a collection of utilities based on the recommendations of residents of Hacker News and GitHub. The list includes: Locust, Vegeta, Slow_cooker, k6 and Siege. They are used by engineers from DICE, EA and Buoyant, as well as the developers of Kubernetes and Load Impact. Let's talk about these tools.

Photos – Victor Freitas – Unsplash

Locust.io

Tool for stress testing sites. All scripts are written in Python. A special web interface built on Flask allows monitoring results in real time. The authors of Locust are Swiss engineers, among whom are employees of DICE and EA companies involved in the development and publication of computer games.

The tool is based on an interesting concept: Locust ("locust") emulates the behavior of a swarm of insects (virtual users), "attacking" the site during the test. Requests are formed using a network library for organizing parallel computing – gevent. Here is an example of a simple test, which is given on the official website of the project:

from locust import HttpLocust, TaskSet, task

class WebsiteTasks (TaskSet):
    def on_start (self):
        self.client.post ("/ login", {
            "username": "test_user",
            "password": ""
        })

    @task
    def index (self):
        self.client.get ("/")

    @task
    def about (self):
        self.client.get ("/ about /")

class WebsiteUser (HttpLocust):
    task_set = WebsiteTasks
    min_wait = 5000
    max_wait = 15000

Locust uses the requests library. This add-on for standard Python tools simplifies working with HTTP and SSL and makes the code more visual. By the way, requests documentation can be used as a cheat sheet for debugging tests on Locust.

This stress testing tool has been around for over seven years. During this time, an extensive community has formed around it – on GitHub more than 10 thousand stars. Locust was used in evaluating the performance of the Battlelog network for the Battlefield series of games. About the tool positively responded Armin Ronacher, author of the Flask framework.

Among the disadvantages of Locust, there is a rather low performance and periodic errors in estimating the response time of sites. Also, the tool does not know how to build graphs, but the problem is solved by unloading the results in the form of CSV files and rendering graphs in the table editor.

If you want to take a closer look at Locust, then you should pay attention to the documentation of the tool. You can also recommend a performance by Alexei Romanov from Wargaming at the Python Meetup. He talks about how to write scripts that emulate user behavior.

Vegetablea

Go command-line utility for testing HTTP services written in Go. It can be connected as a library to create your own load testing tools. Vegeta was developed by one of the authors of the open source Sourcegraph platform, an engine for reviewing and navigating the source code used by Lyft, Uber and Yelp.

Vegeta evaluates the capabilities of network resources, "bombarding" their requests with a set frequency. For example, to check localhost, just enter the following command:

echo "GET http: // localhost /" | vegeta attack -duration = 5s | tee results.bin | vegeta report

By default, Vegeta works with the standard command reading stream (stdin), so the resource for testing is transmitted through echo. The duration parameter indicates the duration of the test. The report will be generated in the results.bin file. Vegeta generates reports in text format, but is able to draw graphs. You can generate them with the following command:

vegeta plot -title = Results results.bin> results-plot.html

A large community has formed around Vegeta – 12 thousand stars on GitHub. The tool was even used by Kubernetes developers to evaluate the performance of their platform – then Vegeta generated about 10 million requests per second to a cluster of thousands of nodes.

The documentation describing all the functions and flags for the Vegeta tests is in the repository on GitHub. You can also find precompiled executable files there.

Slow_cooker

This is a server load testing tool written in Go. It was developed by engineers from Buoyant, which creates a service network for Kubernetes – Linkerd. It is part of the Cloud Native Computing Foundation and is considered a competitor to Google Istio.

Photos – Joshua Aragon – Unsplash

Usually, utilities for load testing check the capabilities of the server, sending it as many requests as possible in a short time. The authors of slow_cooker say that their tool allows you to evaluate the work of iron under a predictable load for a long time.

Buoyant experts use their development to test Linkerd and other services, such as nginx. The instrument is quite young – he is about three – so until he got a large community. But the situation may change in the future, for example, its repository has already been forked at Skysanner, an international service for searching for airline tickets.

You can find the source code on GitHub.

k6

A tool for load and regression testing of microservices, containers, and sites hosted in the cloud. It is written in Go and by JavaScript developers from Load Impact – this is an application for testing the "persistence" of sites.

Work with k6 is based on the everything as code model, when the test logic and all settings are written in JavaScript. In scripts, individual steps can be combined into groups, which may be convenient for those who are used to following the principles of BDD. Here is an example of such a group:

import {group} from "k6";

export default function () {
  group ("user flow: returning user", function () {
    group ("visit homepage", function () {
      // load homepage resources
    });
    group ("login", function () {
      // perform login
    });
  });
};

The tool also allows you to record scripts and build graphs – the last function is implemented on InfluxDB and Grafana. And it has integrations with CI systems like Jenkins, Circle CI, Team City and GitLab.

Users say that k6 is not resource-intensive and has a convenient API. But there are several drawbacks, in particular, k6 does not support websocket and can not conduct tests on distributed systems. Although the k6 developers in the thematic thread on Hacker News said that these features will appear in the future.

If you want to get acquainted with the capabilities of k6 yourself, HN residents recommend starting with technical documentation – it is detailed and with examples. If you have any questions, you can contact the official forum.

Siege

Siege allows you to load test web servers. The utility was created by engineer Jeff Fulmer so that developers can check the resource consumption of their code in conditions close to combat. Siege emulates a continuous stream of access to the site from many users, as if holding the server "under siege" – hence the name of the tool.

After the test, the utility shows: the scan time, the number of transactions per second, throughput, the number of successful and unsuccessful requests, as well as their number with a response code of 200. Here is an example of a report generated by Siege.

Siege is quite widespread in the IT community. For example, a whole section in the book "NGINX High Performance" is devoted to load testing with its help. It is also used by some cloud providers.

Among Siege's drawbacks, non-standard syntax and non-obvious methods for calculating test parameters can be distinguished – for example, redirects are considered successful transactions, so their number may exceed the total number of requests. If you want to try Siege in practice, study the online manual – there are some “oddities” of the system.

Additional reading on 1cloud.ru blog:

What's New in Linux kernel 5.3 – Graphics Drivers, Virtualization, and Other Updates
Why mainstream browser developers again refused to display the subdomain
Why Apple Changed Requirements for Application Developers