Measuring NGFW Performance

The main task of any manufacturer is to create a product that meets the requirements of a potential consumer in order to sell this product to him. And in order to interest the consumer and help with the choice of the right product, manufacturers talk about its characteristics. This applies to any area, from food products to high-tech products such as electronics or cars.

Today we'll be talking about the type of product we understand best – next-generation firewalls (NGFWs).

How do potential consumers evaluate NGFW? One of the main parameters is performance. This parameter is indicated by all manufacturers on their websites and in marketing leaflets. It would seem that everything is simple, the consumer needs to evaluate their tasks and choose a product with suitable performance parameters and the necessary functionality, buy and enjoy. But everything is not so simple, we will explain why.

How to compare NGFW performance based on their description on the website?

NO WAY!

This could be the end of the article, but let's discuss it.

The thing is that each manufacturer uses its own methodology for measuring productivity. And it clearly writes about it. For example:

The manufacturers' methods cover scenarios that are closest to real-life usage conditions, but this is the manufacturer's opinion. Each vendor tests its security gateway on the traffic mix that it considers to be the reference or suitable for testing. The method includes a set of services defined by the manufacturer and specifies their operating parameters.

As a rule, the vendor clearly states in its documents that the results will be different in your conditions, since with any change in conditions – traffic type, enabled functions, settings, number of rules, etc. – the result inevitably changes. This point is clearly stated.

Is it possible to repeat the manufacturer's results on your own?

Yes, you can. If you follow the manufacturer's methodology exactly: create an identical test stand, use identical equipment, use the same software versions, even to the point of matching the assembly build, repeat the settings. Then, with a small error, you can get the result that the manufacturer published. This is an option for those who want to check the vendor. But why do you need such tests if you have completely different conditions and need to solve your own problem?

Why is methodology important?

Let's compare some “hardware” with others. Also technological. Let's consider the importance of measurement methods using the example of the automobile industry.

There are two parameters that almost everyone is interested in when choosing a car, except for the price, of course. These are the maximum speed and fuel (energy) consumption.

It is no secret that in order to determine the maximum speed of a car, manufacturers resort to tricks They use a special method that allows, and sometimes requires, making the car as light as possible (minimum fuel), turning off additional systems that consume energy (air conditioning), conducting tests in ideal weather conditions (no rain, wind, etc.), and the car must move in a straight line without acceleration or braking in the most economical mode.

This is only a portrait of the methodology, but is its purpose and objective clear? Do you see a problem in this? Where is the methodology and where are the real operating conditions?

Let's get back to our NGFWs.

Show what is hidden

Back in the early 2010s, manufacturers' websites published performance figures without disclosing details of how and under what conditions the test results were obtained.

But gradually the market and requirements began to change, now vendors publish performance indicators with mandatory additional explanations. As a rule, now they write something like:

“Performance will vary depending on features activated, and network traffic protocol mix, and packet size characteristics. Performance is subject to change with new software releases. Consult your Cisco representative for detailed sizing guidance”

Each manufacturer has its own methodology and the results of one manufacturer cannot be compared with the results of another until you are sure that the methods and results are comparable.

Naturally, such a variety of results and methods of obtaining them led to the need to create universal and/or independent methods. Let's talk about them.

Universal and independent methods

The request for independent and public testing was accepted and implemented by NSS Labs, an independent organization from the USA that has been researching and testing security solutions since 1991. Most of the world's leading manufacturers in this field submitted their products to NSS Labs for testing, where testing was conducted using its own unified methodology, which allowed the results that NSS Labs published in its reports to be considered comparable.

One of the first widely used techniques, originally intended to measure router performance but successfully applied to testing security gateways, was RFC-2544.

An attentive reader has already noticed the reference to this document in the CheckPoint methodology in the picture above. It is not for nothing that CheckPoint classifies this document as a laboratory methodology or also calls it “ideal conditions”. The methodology involves measuring the speed for the test object with only one rule that passes all traffic.

At some point in time, the global community admitted to itself that the RFC-2544 standard was outdated and the industry needed a new methodology, which resulted in the RFC-9411 document. It was designed for testing security gateways, not routers, and describes many subtle points that are important to consider when testing this type of product.

What methods does InfoTeKS use when testing NGFW and why

In our tests we take the following methods as a basis:

Why do we use NSS Labs methodologies? Because we can reproduce them and compare the results! We use the same Ixia testing tools that NSS Labs used, and we can compare our results with NSS Labs results for other manufacturers.

In addition to the main goals and objectives of our testing – to identify the capabilities of our product and control its quality and stability of operation, we understand the capabilities of our product in comparison with competitors and determine what we need to strive for in order to make it better.

Traffic profile is an important part of the methodology

It is generally accepted that security gateways demonstrate the lowest performance when processing traffic of the smallest IP packet size. Here is what NSS Labs tests show:

As we can see, the maximum performance is achieved for an IP packet of 1500 bytes, the minimum – for packets of 64 bytes. And this is independent of the manufacturer and hardware architecture of the product.

However, thanks to NSS Labs, we have obtained the results of other tests (see below). NSS Labs conducted tests that revealed the throughput of security gateways for cases when one typical type of user traffic is processed. And it turned out that for the SMB protocol, the gateway throughput is even lower than in the case of UDP 64 bytes.

For CheckPoint, the difference is about two times in favor of a 64-byte packet, while for Palo Alto it is almost 17 times.

Let's draw conclusions: the performance of a security gateway depends fundamentally on what type of traffic and what network protocol is processed by the security gateway.

IMIX, EMIX and Customer-MIX

As we found out in the previous section, the behavior – performance of the security gateway – depends on the protocol and traffic type. But the tests that were conducted by NSS Labs show results for only one type of traffic and are simply summarized on one graph for ease of perception.

In real networks, we deal with a variety of heterogeneous traffic. We have so-called “multiservice networks”, that is, networks in which many services operate simultaneously. And each customer has its own set of services in the network.

One of the first manufacturers to think about the problem of identifying bandwidth in multiservice networks was Juniper, which in its publications (https://www.juniper.net/ru/ru/products/security/srx-series/srx4200-services-gateway-enterprise-firewall.html ) began to indicate performance for IMIX (Internet MIX) traffic.

IMIX is an average backbone traffic, to calculate it the community analyzed all Internet traffic (at the time of IMIX development) and identified the share of typical-sized packets in the total volume. It turned out something like this: in one portion of traffic there is 1 packet of 1500 bytes (8.3%), 7 packets of 40 bytes (58.3%) and 4 packets of 576 bytes (33.3%). All packets include the header size. This was probably the beginning, but not the final result, of the consumer struggle to publish comparable performance information.

Yes, IMIX traffic, as in the case of the emergence of fuel consumption indicators for cars in urban and suburban cycles, changed the numbers in the “performance” section on vendors' websites, and buyers began to think about how to interpret such results and apply them to their tasks.

This led to the emergence of a new term Enterprise MIX (EMIX) – a mixture of corporate customers’ traffic. But here a new problem arose: each company had, firstly, its own and unique, and secondly, closed (not public) and no one wanted to share it. And here NSS Labs comes on the scene again with its public methodology.

NSS Labs uses the following traffic mix (see below):

That is, there is a universal method, you can test and compare the results obtained. But even here, it turned out that not everything is as simple as we would like. As was said above, each customer has their own traffic profile, their own conditions, all of this ultimately affects the throughput indicators of the security gateway.

Now the most interesting part. I will give an example from our practice.

A potential customer is testing security gateways. The testing included load testing. Ixia equipment was used for the tests, a testing methodology was developed, and the customer's traffic profile was used.

The test conditions required that a security gateway with a throughput of 1 Gbps be provided. We, in turn, transmitted a gateway, which, according to our measurements, demonstrates the throughput capacity up to 250 Mbit/sec. The potential customer expressed bewilderment about this, but allowed us to test. During the testing process, our security gateway demonstrated a throughput of 920 Mbps.

When the customer asked why we publish the 250 Mbit/sec figure in the official documents, while the device demonstrates significantly better figures, we answered that it was all about the methodology, or more precisely, the traffic profile used in the tests, which was 90% http/https traffic, which is an easy task for security gateways. We use our own methodology and our own traffic profile, as we consider them universal, that is, suitable for most buyers. Experienced readers can develop a discussion on this matter, saying that http/https traffic is “different”, you need to look at the transaction size, whether the so-called SSL decrypt is required, etc. But, in fact, all these nuances are the testing methodology.

Everything is clear, thank you. But what should we do?!

Experts agree that the best way to choose the right security gateway for you is to test it on your traffic under your conditions.

And if this is impossible? Then you need to figure out how the manufacturer got this or that result, that is, get familiar with the testing methodology. And try this very methodology on the real conditions of your network.

Advice from experienced people

Our recommendations on what to pay attention to:

Number of rules

When performance data is published, it is rare that anyone specifies how many filtering rules were active at the time of testing. For example, CheckPoint states 100 filtering rules in its SPU (SecurityPowerUnit) methodology.

Is this enough? It would seem so, because if testing is carried out according to RFC-2544, then the filtering rule in the test is one – everyone is allowed to do everything. And here it is 100 times more.

But! RFC-9411 assumes the use of 562 filtering/access rules. 5 times more! Maybe this is enough for testing?

There is no correct answer. We have a customer who had 5,000 filtering/access rules at the start of the project, and a year later there were 10,461.

What should the customer do? Is the number of rules important during testing and then in operation? As experience shows, the number of rules is an important parameter and it seriously affects the performance of the security gateway.

If the manufacturer states that testing is carried out according to the RFC-2544 methodology, then you understand that this is one rule. Most likely, this is not enough for you. If RFC-9411, then there are significantly more rules – 562. This is probably enough, at least this is what all information security practitioners teach us, saying that it is necessary to optimize the information security policy.

But if you understand that you have your own case, then you need to discuss it with the manufacturer.

Traffic profile

Let's go back to the traffic profile. We mentioned that the same security gateway can demonstrate different performance depending on the type of traffic. And when there is a mixture of traffic, the result can only be determined by testing.

NSS Labs tests contain gateway performance results depending on the type of traffic. They show that gateways demonstrate high performance when processing FTP, but show a result 10 times lower when processing SMB traffic. And it is clear that this behavior is typical for gateways of all manufacturers.

Next, we look at the EMIX mix from NSS Labs and see that this mix contains only 5% FTP and no SMB at all. And we have a customer whose traffic is almost 50% SMB. Obviously, the security gateway in the case of this customer will show low performance indicators, but the fault lies not with the gateway manufacturer, not with the methodology, but with the SMB traffic itself.

The best solution is to create your own traffic profile and ask the manufacturer to test it. If this is not possible, then try to understand what traffic profile the manufacturer uses for its tests, how it compares to yours, and try to approximate this data. Quite a task…

Journaling

When analyzing performance data, no one asks whether logging is enabled. And as practice shows, the task of logging everything that happens in a security gateway is quite resource-intensive. This is taken into account in the RFC-2544 methodology, as well as in RFC-9411. However, the first methodology does not define the action, which means that logging during testing may not be enabled, while the second requires enabling logging, but does not describe the level of detail. Therefore, if the vendor declares that the tests are carried out according to RFC-9411, then logging is enabled, but the level of detail is unknown. And if according to RFC-2544, then logging is definitely disabled. If you use your own methodology, you need to find out the details from the manufacturer. At the same time, do not forget that the customer's requirements usually include logging everything!

Conclusions

If by the end of the article you haven’t yet drawn your own conclusions, here’s a hint:

1. All manufacturers are honest with their customers.

2. The parameters published by the manufacturer were obtained using their own product testing methods.

3. All manufacturers describe their methodology in detail, search for or request a document.

4. It won't help you anyway, because the product will show completely different numbers in your conditions, and that's normal.

5. Before purchasing a product, contact the integrator or vendor to have them test it under conditions as close to your actual conditions as possible.