Protecting the service from overload using HAProxy

If you have ever used HAProxy for traffic balancing, you've probably at least heard that this product can track service and user activity indicators and respond to them according to predefined conditions. Typically, articles on this topic provide an example of limiting a user by source IP address if the frequency of requests from him exceeds a certain predetermined limit. Herefor example, this article from the developers’ website.

I would like to delve a little deeper into the topic of the stick tables mechanism used, but talk not about users who are actively interested in your site, but about the load capacity, or capacity, of the entire site (or some of its paths). Firstly, any service is limited in the number of simultaneous requests that can be served using existing resources. Secondly, most often the service has more than one platform or at least more than one instance of the balancer. This means that catching a lone user is, of course, great, but I would also like to solve another interesting problem: protecting the service from overload in general and in the event that there is more than one balancer. As a bonus, we’ll talk about the problem of smart load redistribution between locations.

Our service

So, let's assume that we have made a news display service. There is a lot of news, they are all very fresh. We want to give our clients the very latest and hottest news so much that we don’t even use cache. The load, to a certain extent, linearly depends on how many clients come to us and how often they request site pages.

Here is our site, codenamed Alpha. We have three servers (server01a, server02a, server03a) that constantly and asynchronously receive news from some secret source. There is a traffic balancer balancer01a based on HAProxy, which distributes the client load between three servers and makes sure that they are alive and responding to users.

Let's assume that the servers do not have any hidden limitations or dependencies and their performance depends only on the hardware used or the type of virtual machine. We conducted load tests and found that these particular servers are capable of generating and delivering to users two hundred pages per second each. We assumed that the servers need to be maintained, and the site is capable of working with peak load without one server. Thus, the load capacity of the Alpha site is currently 400 user requests per second, or 400 RPS (requests per second).

Let's write a simple config for HAProxy that matches our picture:

frontend fe_news
  bind :80
  default_backend be_news

backend be_news
  default-server check
  server server01a 10.0.0.11:80
  server server02a 10.0.0.12:80
  server server03a 10.0.0.13:80

The fe_news frontend section defines parameters facing the user, and the be_news backend section defines parameters facing the servers.

As you can see, there is nothing in the configuration file that would protect the site from the influx of visitors. Let's fix this.

Just in case, let me remind you that HAProxy allows you to limit each server and section by the number of simultaneous requests. If this is your scenario, please use it.

Restricting individual users is of little use to us, since a news wave can bring many new visitors who will come only once. And these total requests are quite enough for us to not be able to cope. In our case, it is necessary to count the number of simultaneous requests to the site (request rate) and do something with those that are above the plan.

To implement this plan, we will need stick tables. I'll talk a little about them here, and you can find out more about them in documentation.

Stick tables

Stick table in HAProxy is a cache-like key-value database in application memory in which various counters can be stored in relation to some primary key. The key can be an IP address, a number, or an arbitrary fixed-length string. What a table has in common with a cache is that a fixed time is set for each table, after which a non-updating row disappears. The table is always bound to a proxy section (frontend, backend, listen). A section cannot have more than one table.

There is another way to declare a stick table – this is the peers section, but we will talk about this later.

Example stick table declaration:

frontend fe_news
  bind :80 
  stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)

Please note that the key type IPv6 can also store IPv4 addresses by converting them to IPv6 format. The field sizes are also in the documentation. For IPv6 keys, this is approximately 60 KB per entry.

In this particular case, we store the user's HTTP request rate for the last 10 seconds (http_req_rate(10s)). In essence, it doesn’t matter whose addresses—the client’s or the server’s—are stored. But the servers would hardly need a million records (size 1m) and 10-second expiration period (expire 10s). Counters can be quite varied, and there may be more than one. There is detailed documentation on this topic Here.

Sticky counters

The table itself is useless; you need to store something in it. For this purpose, HAProxy has so-called sticky counters. Sticky counter is a rule in the proxy section that associates a specific attribute with the specified stick table. By default, for the non-commercial version of HAProxy there are three such rules per request, but their number can be increased by a global directive tune.stick-counters. Each rule adds 16 KB to the connection information structure and another 16 KB to each request. Binding is carried out using the track request or response method. Example:

frontend fe_news
  bind :80
  stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
  http-request track-sc0 src

In this example we are using sticky counter sc0to track the frequency of HTTP requests from each IP address for the fe_news proxy section in the stick table local to this section.

ACL expressions

Based on the values ​​monitored using sticky counters, you can perform certain actions on traffic or simply calculate useful statistics for logs. Or, for example, this is how you can start blocking (return 403) the IP address of a client from which there are too many requests per unit of time to the main page /:

frontend fe_news
  bind :80
  stick-table type ipv6 size 1m expire 10s store http_req_rate(10s)
  http-request track-sc0 src if { path / }
  http-request deny if { sc0_http_req_rate() gt 100 }

Please note that the sc0 counter will increase on both successful and unsuccessful attempts. That is, to unlock, it is necessary that the client make less than 100 attempts to open the page / in the last 10 seconds, otherwise he will receive a permanent 403 error.

Protecting the site from overload

We've introduced ourselves to stick tables, and now let's get back to our task. Let me remind you that we want to limit traffic to the site to 400 requests per second and we don’t care whether it’s one client or a million. Here's how we'll change the frontend section:

frontend fe_news
  bind :80
  default_backend be_news
  stick-table type integer size 1 expire 1s store http_req_rate(1s)
  http-request track-sc0 always_true
  http-request deny if { sc0_http_req_rate() gt 400 }

What's going on here:

1. For our main frontend section, we declared a stick table of type “integer” with one single row from which we want to know “the number of requests in the last second.”

2. For each user request, we update that single row in the table using the predefined constant always_true, which is converted to some kind of integer. It doesn’t matter to us which one (although it will be one) – as long as it’s the same. This way we update the http_req_rate counter for the table.

3. If the total request counter has exceeded 400 in the last second, we return a 403 error to the user.

Is everything okay with this decision?

Remember when I said that the counter would be triggered for both successful and unsuccessful requests? This means that once our site reaches 400 requests per second, it will start returning a 403 error to all clients. Additionally, for overload situations, there is a better response code than 403: 429 (Too Many Requests).

Let's fix the solution. For this we need variables. There are a few words about them in the official documentation.

frontend fe_news
  bind :80
  default_backend be_news
  stick-table type integer size 1 expire 1s store http_req_rate(1s)
  http-request track-sc0 always_true
  http-request set-var(req.rps) sc0_http_req_rate()
  http-request set-var(req.rnd) rand,mod(req.rps)
  acl rps_over_limit var(req.rps) -m int gt 400
  acl unlucky_request var(req.rnd) -m int ge 400
  use_backend be_error_429 if rps_over_limit unlucky_request

backend be_error_429
  errorfile 429 /etc/haproxy/errors/429.http
  http-request deny deny_status 429

Let's figure it out:

1. We still count the number of requests per second (RPS) in the stick table.

2. Next, we add the current RPS value into the req.rate variable. Then in the req.rnd variable we put the remainder of dividing the random number by the current request rate.

3. Then we add two ACLs. The first of them (rps_over_limit) is triggered if the number of requests per second exceeds the limit of 400. The second (unlucky_request) is triggered when the value of the random number is also higher than or equal to this limit.

4. If both ACLs succeed, we return an error. The logic here is this: suppose the current load on the site is 600 RPS. This means that ACL rps_over_limit will always work. The random number generator will return us values ​​in the range from 0 to 599. The threshold value is 400. Thus, in 33% of cases, the unlucky_request ACL will work and the user request will be rejected. But in the remaining 66%, unlucky_request will not work and the client will receive his content.

5. We want to present the error beautifully and therefore return it with a separate proxy section be_error_429. In it, we determine which specific template to use for the error page and return this pre-prepared page. By the way, instead of an error, you can give something else: for example, a cached copy of the content or some kind of game to distract the user from reading the news.

Second balancer

Suddenly we remember that the balancer also lives on the server, which means that it also needs to be maintained sometimes. That is, we at least need one more instance of HAProxy.

How can we protect the site now?

Halve the limits to 200 RPS per balancer? Bad: on the one hand, the distribution of traffic between balancers may be uneven, on the other hand, when working on one of the balancers, we will sag in capacity.

Use distributed traffic counter for coordination? Great idea. But here's the problem: in the case of HAProxy, it is part of the commercial version of the product (Global Profiling Engine). If you can afford a commercial subscription, then this is the best way. With alternatives, everything is also not easy, since it must be a good free distributed counter that integrates well with HAProxy, is publicly available and supported by developers. If you know one, please write about it in the comments.

For now, let's use what we already have and try to implement a distributed counter. To do this, we will need HAProxy's distributed synchronization capabilities for stick tables.

Distributed synchronization stick tables

HAProxy has the ability to actively-standby replicate stick tables from one node to many others. The main thing to remember about this process is that it is one-way.

If you write something to a replicated table on one host, that change will overwrite the contents of the tables on the remaining hosts. It will just overwrite, not add. It is important.

If one of the synchronization participants restarts, it will automatically pull the contents of the tables to itself. This has one nice side effect: restarting the local HAProxy causes the new instance to pull all the contents of the tables from the old one, so it is not lost. This means that even if you declare a synchronization group for servers from the same local host, this will already bring some benefit.

To configure synchronization, HAProxy has a special peers section, in which all exchange participants are declared, and stick tables can also be declared. Remember when I said that in addition to proxy sections, they can be declared in one more way? Moreover, multiple stick tables can be declared in the peers section. A change to any of them will cause notifications to be sent to all participants.

Let's look at an example for our new site topology for balancer01a:

global
  localpeer balancer01a

peers news_peers
  bind :10000
  # You can use SSL as well:
  # bind 10.0.0.1:10000 ssl crt mycerts/pem
  # default-server ssl verify none
  server balancer01a # local peer
  server balancer02a 10.0.0.2:10000
  table balancer01a type integer size 1 expire 3s store http_req_rate(3s)
  table balancer02a type integer size 1 expire 3s store http_req_rate(3s)

Both balancers of the Alpha platform are included in the synchronization group. In addition, in this group we will immediately declare two tables that we will use: one on the balancer01a host, and the second on balancer02a. This kind of setup will require a unique configuration file for each of the balancer instances, but this is easily solved using templating (Jinja or any other templating engine you like).

Pay attention to the localpeer parameter of the global section. It specifies the identifier by which HAProxy will select a local server in the peers section. By default, the server hostname will be used, but it is better not to risk it and explicitly set the value.

Also note that I increased the request_rate period from one second to three. This is done in order to smooth out traffic spikes that may appear due to some delays in synchronization.

So we have two tables. Let's modify the frontend section so that it uses new tables and takes into account changes on the neighboring balancer. At the same time, we will take into account the change in the interval for calculating the average number of requests.

Here is the configuration for balancer01a. I show in bold what has changed:

frontend fe_news
bind:80
default_backend be_news
http-request track-sc0 always_true table news_peers/balancer01a
http-request set-var(req.rps01a) sc0_http_req_rate(news_peers/balancer01a),div(3)
http-request set-var(req.rps02a) sc0_http_req_rate(news_peers/balancer02a),div(3)
http-request set-var(req.rps) var(req.rps01a),add(req.rps02a)
http-request set-var(req.rnd) rand,mod(req.rps)
acl rps_over_limit var(req.rps) -m int gt 400
acl unlucky_request var(req.rnd) -m int ge 400
use_backend be_error_429 if rps_over_limit unlucky_request

For balancer02a the section will differ by one line:

frontend fe_news
http-request track-sc0 always_true table news_peers/balancer02a

If we have more than two balancers, we will have to add more than two tables. But, again, automatic generation of configurations based on templates closes this issue.

Second site

Our media empire is growing, and more and more users are reading news from us. Things are going so well that we decided to launch another platform – Beta.

However, we don’t know which site the user will come to: what if most of it will be routed to Alpha and there will be an overload with 429 errors in the direction of users, while Beta will be idle? What to do?

Maybe connect all servers for all balancers and distribute traffic evenly? Can. But traffic between locations costs money, and I wouldn’t want to waste it if local servers are idle.

How about we send traffic to a neighboring site only when the local one is already loaded? In addition, we know that each server pulls 200 RPS of load. Why do we count 400 RPS per location? Let's start with the number of live servers. Forward!

Disclaimer: I will provide configurations for balancer01a, for everyone else everything will be similar, as in the previous example.

So, first we need to fix the peers section. What has changed is highlighted in bold:

peers news_peers
bind:10000
server balancer01a # local peer
server balancer02a 10.0.0.2:10000
server balancer01b 10.0.1.1:10000
server balancer02b 10.0.1.2:10000
table balancer01a type integer size 1 expire 3s store http_req_rate(3s)
table balancer02a type integer size 1 expire 3s store http_req_rate(3s)
table balancer01b type integer size 1 expire 3s store http_req_rate(3s)
table balancer02b type integer size 1 expire 3s store http_req_rate(3s)

There's nothing interesting here other than the obvious changes. The only thing I would strongly recommend in real life is to enable SSL, both for synchronization and communication with servers.

But now it gets more interesting. First, we need to redo the backend sections to add new servers there:

backend be_news_local
  server server01a 10.0.0.11:80 track be_news/server01a
  server server02a 10.0.0.12:80 track be_news/server02a
  server server03a 10.0.0.13:80 track be_news/server03a

backend be_news
  default-server check
  server server01a 10.0.0.11:80
  server server02a 10.0.0.12:80
  server server03a 10.0.0.13:80
  server server01b 10.0.1.11:80
  server server02b 10.0.1.12:80
  server server03b 10.0.1.13:80

Now we have two sections: one contains only the servers of the current location, the second contains everything. In order not to check the availability of the same server twice, we will use the track option, which tells the balancer: take the status of this server from the status of that server from that proxy section.

Now let's look at the frontend section:

frontend fe_news
  bind :80
  default_backend be_news

  http-request track-sc0 always_true table news_peers/balancer01a
  http-request set-var(req.rps01a) sc0_http_req_rate(news_peers/balancer01a),div(3)
  http-request set-var(req.rps02a) sc0_http_req_rate(news_peers/balancer02a),div(3)
  http-request set-var(req.rps01b) sc0_http_req_rate(news_peers/balancer01b),div(3)
  http-request set-var(req.rps02b) sc0_http_req_rate(news_peers/balancer02b),div(3)
  http-request set-var(req.rpsa) var(req.rps01a),add(req.rps02a)
  http-request set-var(req.rpsb) var(req.rps01b),add(req.rps02b)
  http-request set-var(req.rps) var(req.rpsa),add(req.rpsb)
  http-request set-var(req.local_srv_count) nbsrv(be_news_local)
  http-request set-var(req.srv_count) nbsrv(be_news)
  http-request set-var(req.avg_local_srv_rps) var(req.rpsa),div(req.local_srv_count)
  http-request set-var(req.avg_srv_rps) var(req.rps),div(req.srv_count)
  http-request set-var(req.rnd) rand,mod(req.avg_srv_rps)

  acl local_rps_ok var(req.avg_local_srv_rps) -m int le 200
  acl local_srv_alive var(req.local_srv_count) -m int gt 0
  acl rps_over_limit var(req.avg_srv_rps) -m int gt 200
  acl unlucky_request var(req.rnd) -m int ge 200

  use_backend be_error_429 if rps_over_limit unlucky_request
  use_backend be_news_local if local_srv_alive local_rps_ok

Let's look at the configuration:

1. We calculated the RPS for each balancer and added the results into the variables req.rps01a, req.rps02a, req.rps01b and req.rps02b.

2. Then we calculated separately the RPS for the Alpha site and the Beta site and put the results in req.rpsa and req.rpsb.

3. We calculated the total RPS for all locations and put it in req.rps.

4. The nbsrv() function returns the number of live servers from the specified proxy section. We used it to calculate the average RPS per site-local Alpha server (req.avg_local_srv_rps). Then do the same for all service servers (req.avg_srv_rps).

5. Then we generated a random number in the range from 0 to the average RPS per server and stored the number in (req.rnd).

6. Then a series of conditional commands begins:

   • if the average RPS to the service server is greater than 200 and the random number is greater than or equal to 200, return an error to the user;

   • if local servers are alive (at least one) and the total load on them with traffic local to the site is normal (less than 200 per server on average) – use the be_news_local proxy section, that is, service the request locally (use_backend be_news_local);

   • otherwise, send the request for processing to a random live server of the service (default_backend be_news).

So what we got:

• If the load is normal – no more than 200 RPS on average per local server – the user will be served by a server on the same site where he came.

• If the load is slightly higher than the norm for a site (say, 610 RPS for a site with three live servers), but overall it is acceptable for the service, the request will be served by a random server from one of the two sites.

• If the load is higher than the maximum allowable (200 × number of live servers), then user requests will be partially discarded in order to meet the allowed level on average.

  If N is the number of live servers, CurrentRPS is the current traffic, and MaxServerRPS is the maximum RPS that can be served by the server, then the probability of the user receiving an error is:

  

Thank you for reading to the end of the article! I tried to show how stick tables with distributed synchronization are a powerful and useful component of HAProxy. Unfortunately, I haven't seen much documentation on this issue. Even the syntax for working with variables is described very sparingly in the official documentation. Therefore, I hope that the article will be useful to you. Of course, in real conditions, you will have to separately calculate the load on different “handles” of the service, and the backends will have different capacities. But for the first, HAProxy has an ACL, and for the second, there are server weights. In general, everything will work out!

As you grow, so does your project. To grow, you need to step out of your comfort zone and take a step towards change. You can learn new things by starting with free classesor open up prospects with training “Network Engineer” With promo code CODEHABR10 the price is even nicer.