A Beginner's Guide to Load Testing
In this guide, we will learn: why to conduct load testing, what it depends on, what is the architecture, what approach will ensure its successful implementation, how to set up the environment, best practices, as well as current tools for load testing available on the market.
We are familiar with the division of testing into functional and non-functional. Non-functional testing includes performance testing, security testing, user interface testing, and so on. Load testing is a type of performance testing.
When testing an application for performance, what exactly do we check? In this case, we test the application for load, volume, capacity, stress, and other parameters.
What is load testing?
Load testing tests the system's response to various load conditions, including simulating multiple users accessing the application at the same time. This type of testing typically measures the speed and performance of the application.
Thus, whenever we change the load, we observe the behavior of the system under different conditions.
Example: Let's say a client requirement for a login page is a response time of 2-5 seconds, and these 2-5 seconds should remain stable under a load of up to 5,000 users. What should we be looking at here? Is it just the system's ability to handle the load, or is it just the response time requirement?
The answer is both. We need a system that can handle a load of 5,000 users with a response time of 2-5 seconds for all concurrent users.
What is meant by simultaneous and virtual users?
Concurrent users are those who log into an application, perform a set of actions at the same time as other users, and log out of the application at the same time. Virtual users, on the other hand, simply log in and out independently of other users.
Load Testing Architecture
In the diagram below, we see how different users access the application. Here, each user sends a request over the internet, which then passes through the firewall.
After the firewall, requests are sent to a load balancer, which distributes the load between the web servers, after which the requests are sent to the application server, and then to the database server, where the necessary information is retrieved according to the user's request.
Load testing can be done manually or using tools. However, manual load testing is not recommended because we do not test the application under light load.
Example: Let's say we want to test an online shopping app to see the response time of the app for each user click. So Step 1 is launching the URL, measuring the response time, logging into the app and measuring the response time, then selecting a product, adding to cart, paying, and logging out. All these steps need to be done for 10 users.
If we want to load test an application for only 10 users, we can do it by manually creating the load with 10 physical users on different machines instead of using a tool. In this case, it makes sense to do the load testing manually instead of investing in a tool and setting up an environment for it.
However, imagine that we need to test the load for 1500 users. In this case, it is necessary to automate the load testing using one of the available tools, based on the technologies on which the application is built, as well as the budget allocated for the project.
If we have budget then we can use commercial tools like Load runner but if budget is limited then we can use open source tools like JMeter etc.
Whether a commercial or open source tool is used, all the details need to be discussed with the client before the final selection of the tool. Typically, a proof of concept (PoC) is prepared where a sample scenario is created using the selected tool and sample reports are shown to the client for approval of the tool before finalization.
In automated load testing, we replace users with an automation tool that simulates user actions in real time. By automating the load, we can save resources and time.
The diagram below shows how users are replaced using the tool.
Why do you need to perform load testing?
Let's say you have an online store website that works well on normal business days – that is, users can log into the app, browse through different product categories, select products, add products to their cart, place an order, and log out without encountering any errors or lags.
But then comes a peak day, say Black Friday, and thousands of users start logging into the system. Suddenly, the system crashes – and users experience a very slow response, some can't even log into the site, some can't add an item to the cart, and some can't place an order.
On this significant day, the company suffers huge losses, losing many customers and payments. All this happened only because they did not predict the user load on peak days, and even if they had predicted, the company's website had not been load tested, so they did not know what load the application could withstand on peak days.
Thus, to avoid such situations and minimize revenue losses, it is recommended to perform load testing for such types of applications.
Load testing helps create reliable and stable systems.
Bottlenecks in the system are identified early, even before the application goes into production.
Load testing also helps determine the capacity of an application.
What does load testing achieve?
Properly conducted load testing allows you to accurately understand the following:
The number of users the system can support or scale to.
The response time of each transaction.
How do individual system components behave under load, such as application server components, web servers, databases, etc.
Which server configuration can handle the load best?
Is the current equipment sufficient or is additional equipment required?
Bottlenecks such as CPU and memory usage, network latency, and so on.
Wednesday
To perform the tests, we need a special load testing environment. This is important because the load testing environment is often similar to the production environment, and the data in this environment is similar to production, although not completely identical.
There are different test environments such as SIT (System Integration Testing) environment, QA environment, etc. – they are not like production environment because unlike load testing, functional or integration testing does not require as many servers or test data.
Example:
In the production environment, we have 3 application servers, 2 web servers and 2 database servers. In the QA environment, we only have 1 application server, 1 web server and 1 database server. Therefore, if we perform load testing in a QA environment that is not equivalent to the production environment, then our tests will be invalid and incorrect, and the results of such tests cannot be used.
So always try to have a separate environment for load testing, similar to production.
Additionally, sometimes our system calls into third party applications, and in those cases stubs can be used since we cannot always work with external providers to update data or resolve other issues and get support.
Try to take a snapshot of the environment when it is set up so that if you need to rebuild the environment, you can use this snapshot, which will save time. There are several tools in the market to create the environment, such as Puppet, Docker, etc.
Approach
Before you begin load testing, you need to understand whether the system has been load tested before. If it has, you need to find out what the response time was, what client and server metrics were collected, what the user load was, and so on.
Information about the current processing capabilities of the application is also required. If it is a new application, you need to understand the requirements: what is the target load, the expected response time, and whether these indicators are achievable.
If it's an existing application, you can get load requirements and user access patterns from the server logs. In the case of a new application, you'll need to contact the business team for all the information you need.
Once you have the requirements, you need to determine how the load testing will be performed. Manually performing a load test is resource-intensive and very expensive. In addition, repeating the test over and over again will also be difficult. Therefore, if the project budget allows, it is worth choosing commercial tools: they have many features, support many protocols, and are usually very easy to use.
Our approach to load testing will be as follows:
1) Defining the acceptance criteria for the load test
For example:
The login page response time should not exceed 5 seconds even under maximum load.
CPU load should not exceed 80%
The system throughput should be 100 transactions per second.
2) Identify business scenarios that need to be tested
Don't test all flows. Instead, try to understand the core business processes that are expected in production. If this is an existing application, this information can be obtained from server logs in production.
If it's a new application, you'll need to work with the business teams to understand usage patterns and workflows. Sometimes the project team will hold workshops to give an overview or details of each component of the application.
It is necessary to participate in such seminars and record all the necessary information to conduct load testing.
3) Workload modeling
After obtaining detailed information about the business processes, user access patterns, and number of users, we need to design a workload model in such a way that it simulates the actual user behavior in production or as it is expected in the future when the application goes live.
When designing a workload model, it is important to keep in mind the time it will take to complete a particular business process. It is important to set the right think time so that the user navigates the application in the most realistic way.
The workload pattern typically includes ramp up, ramp down, and steady state. The system needs to be stressed slowly, so ramp up and ramp down tempos are used. The steady state is typically an hour-long load test with a 15-minute ramp up and 15-minute ramp down.
Let's look at an example workload model:
Application Description – Let's say this is an online store where users go into the application and can choose from a variety of products, such as clothing, by navigating through each product.
To view detailed information about each product, the user needs to click on it. If he is satisfied with the price and appearance of the product, he can add it to the cart and buy it, completing the payment process.
Below is a list of scenarios:
Browsing – At this stage, the user enters the app, browses through the different categories, and exits the app.
View, view product, add to cart – the user logs into the app, browses through different categories, views product details, adds the product to the cart, and logs out of the app.
Browse, View Product, Add to Cart, and Checkout – In this scenario, the user logs into the app, browses through different categories, views product details, adds the product to the cart, checks out, and logs out.
Browse, view product, add to cart, checkout and pay – In this case, the user logs into the app, browses through different categories, views product details, adds the product to the cart, checks out, pays and logs out.
No. | Business process | Number of transactions | Virtual user load | Response time (sec) | % Acceptable Failure Rate | Transactions per hour |
1 | View | 17 | 1600 | 3 | Less than 2% | 96000 |
2 | View, view product, add to cart | 17 | 200 | 3 | Less than 2% | 12000 |
3 | Browse, view product, add to cart and checkout | 18 | 120 | 3 | Less than 2% | 7200 |
4 | View, view product, add to cart, checkout and payment | 20 | 80 | 3 | Less than 2% | 4800 |
The above values were calculated based on the following calculations:
Transactions per hour = Number of users * Transactions made by one user in one hour.
Number of users = 1600.
Total number of transactions in the “View” scenario = 17.
Response time for each transaction = 3.
Total execution time of 17 transactions by one user = 17*3 = 51, rounded to 60 sec (1 min).
Transactions per hour = 1600*60 = 96000 transactions.
4) Development of load tests — load tests should be designed taking into account the data we have collected so far — business processes, number of users, user patterns, and metrics that need to be collected and analyzed. In addition, the tests should be as close to reality as possible.
5) Performing a load test — Before running a load test, make sure the application is up and running. The load testing environment is ready. The application is functionally tested and stable.
Check the load testing environment configuration settings. It should be identical to the production environment. Make sure all test data is available. Don't forget to add the necessary counters to monitor system performance during the test.
Always start with a small load and gradually increase it. Never start with a full load, so as not to disrupt the operation of the system.
6) Analysis of load testing results — Have a baseline test at your disposal so that you can always compare others to it. Collect metrics and server logs after the test to identify bottlenecks.
Some projects use Application Performance Monitoring (APM) tools to monitor the system while running tests. These tools help to identify the root cause more easily and save a lot of time. APM tools provide a broad overview, allowing you to pinpoint where the problem is.
Some of the APM tools available in the market include DynaTrace, Wily Introscope, App Dynamics, and others.
7) Reporting — after completing the testing, collect all the metrics and send a summary report with the testing results to the relevant team with your comments and recommendations.
Best practices
Always check the stability of the application before running load testing. The functional testing team should confirm the functional stability of the application, and all major defects should be fixed and tested before the build is copied to the load testing environment.
Ensure that the load testing environment is a replica of or close to the production environment, including the number of servers, load balancers, server configurations, and firewalls.
Ensure that test data is unique and that all test data is copied to the load testing environment before starting load testing.
Design test cases so that they simulate real user actions that occur in a production environment.
Design the workload based on the production user load and business processes. For a legacy application, if it is new to you, work with the business team to clarify the business processes and user load.
Collect all important metrics such as response time, requests per second, throughput, CPU utilization, memory, network, and the number of running virtual users (Vusers).
Read more: How I scaled an Amazon load generator to run on 1000 machines.
We invite everyone interested in the topic of load testing to the upcoming open lessons:
