Performance Testing for iOS
We all love it when the application we use works responsively, quickly, and also when the operations we want to perform happen as quickly as possible: be it a banking application, a commercial application, etc.
But how can we track and measure the metrics of our application speed? This is a question that many developers and companies ask themselves when they have received negative feedback or think ahead when the code base and complexity of the application will grow. There are two ways: either we study on real users of our application, or we look for some other way that allows us to statistically test the hypothesis.
Solutions on real users
Method 1: XCodeOrganizer
Xcode Organizer is used to analyze app startup metrics, interface responsiveness, memory and energy consumption, and other important aspects. This tool allows you to sort data by device models, app versions, and user groups. Xcode Organizer helps you determine whether performance has improved due to optimizations you've made.
Method 2: MetricKit
MetricKit is a framework that serves as a tool for monitoring application performance. It serves to aggregate information into histograms, such as network status, initialization duration, and other performance-related parameters. Developers also have the ability to add their own measurements based on OSSignpost events in the application.
import MetricKit
class MySubscriber: NSObject, MXMetricManagerSubscriber {
var metricManager: MXMetricManager?
override init() {
super.init()
metricManager = MXMetricManager.shared
metricManager?.add(self)
}
override deinit() {
metricManager?.remove(self)
}
func didReceive(_ payload: [MXMetricPayload]) {
for metricPayload in payload {
// Do something with metricPayload.
}
}
}
Method 3: TestFlight
If you use TestFlight as a solution for both internal and external testing, there is a form inside, by filling out which we can receive complaints not only about crashes, but also about lags that interfere with interaction with the application.
A little bit of synthetics
But all of the above only works on real users. What if we want to test changes before end users see them (for example, database migration, or a major change to the root library).
The simplest answer would be: using tests. Because if you think about it:
when we want to be sure that part of our code works and will work correctly, we write unit tests (XCTests).
When we want to check that our application works correctly and will continue to work, we write integration tests (XCUITests).
when we want to check that the layout in our application looks and will look the way we need – we write snapshot tests. (FBSnashopt as an example). The speed application is no exception – we write performance tests. And what is it?
Performance testing of mobile applications is a type of software testing that evaluates the speed, responsiveness, stability, and overall performance of a mobile application under various conditions.
The main goal of performance testing is to ensure that the application works well and provides a positive user experience across different devices, networks, and usage scenarios. It helps developers identify and eliminate performance bottlenecks, optimize resource usage, and improve the overall behavior of the mobile application.
Examples of metrics
There are several metrics that are particularly valuable in the application:
Application Launch Metrics – besides the fact that if this metric is too large, the system can stop the application at the moment of launch, it is important because at the moment of launch the user cannot perform any action in the application and can easily think that the application has frozen and go to a competing application. Example: Startup Metrics or TTI (Time to interactive), which can be seen not only in mobile development, but also on the web.
Resource consumption metric – we have big limitations inside our application: energy efficiency, memory consumption, disk reading, CPU usage, etc. In small applications this may not be so obvious, but as soon as the code base, complexity, functionality grows – each of the aspects will be critical.
Application performance metric – characterizes how fast certain parts of our application work. For example, Hitch Ratio – the number of frames lost per unit of time (especially relevant for devices with 120 Hz screens). Also, do not forget about the network layer, but not from the data loading side (very relative to the network capabilities), but data decoding and conversion.
Metrics collection
1. Via measure
The easiest way to collect metrics that we can collect in our tests is through a block measure. Call this method from a test method to measure the performance of a code block. By default, this method measures the number of seconds it takes for the code block to execute.
final class LaunchPerfTests: XCTestCase {
override func setUp() {
super.setUp()
let app = XCUIApplication()
app.launch()
}
func test_launchPerformance() throws {
measure(metrics: [XCTApplicationLaunchMetric()]) {
XCUIApplication().launch()
}
}
}
An example of the simplest application launch test.
Also, for example, if we want to see how long methods take to execute before the user can finally use our application (analogous to the Time to interactive metric), we can use a similar test as:
final class LaunchPerfTests: XCTestCase {
override func setUp() {
super.setUp()
let app = XCUIApplication()
app.launch()
}
func test_launchPerfUntilResponsive() throws {
let app = XCUIApplication()
measure(
metrics: [XCTApplicationLaunchMetric(waitUntilResponsive: true)]
) {
app.launch()
app.activate()
}
}
}
An example of a test to launch an image before interactivity.
The above test uses one of the basic metrics, such as the startup metric. It can be especially relevant when, for example, you make changes to module linking or move some part of the application startup later, etc. However, do not forget to set the baseline settings in order to understand in which segment our results should be.
There are also other metrics that we can write down by default:
XCTCPUMetricto record information about processor activity during a performance test.XCTClockMetricto record the time elapsed during the performance test.XCTMemoryMetricto record the physical memory used in performance testing.XCTOSSignpostMetricto record the time it takes for a performance test to execute a designated area of code.
2. Through signposts
But what if we want to track some other metrics, like hitch ratio, etc.? Pointers come to our rescue. They allow us to record useful debugging and analysis information and include dynamic content in our messages.
There are two ways to create signposts:
Old API via os_signpost.
New API, via
OSSignposter(We will consider it).
let signposter = OSSignposter()
let signpostID = signposter.makeSignpostID()
let data = fetchData(from: request) // Создание события, чтобы отметить определенную точку интереса.
signposter.emitEvent("Fetch complete.", id: signpostID) processData(data) // Завершение указанного интервала, используя сохраненное состояние интервала.
signposter.endInterval("processRequest", state)
Next we create an instance of the class XCTOSSignpostMetricwhich accepts information about our created signpost above:
func signpostMetric(for name: StaticString) -> XCTOSSignpostMetric {
return XCTOSSignpostMetric(
subsystem: subsystem,
category: category,
name: String(name)
)
}
And after that we put the new metric into the block measure(metrics: ...)which we discussed a little earlier.
3. Common space
We've already looked at examples using signposts and pre-implemented methods for collecting metrics, but what if we want to collect metrics that we can't apply a rule with a start and end point to? Or we need some custom rule for calculating metrics: our own hitch ratio metric, etc. The question arises: how to collect this metric? After all, all speed tests use XCUITests by default. And we can't directly transfer data from Runner (the application that contains the tests) to Target (the main application) and vice versa.
Here we come across a technique that we use when writing UI tests and communication between two applications, for example, for deep link transitions.
import Foundation
import Network
/// Класс в target приложении, который отвечает за отправку событий
final class DataSender {
private var host: NWEndpoint.Host?
private var port: NWEndpoint.Port?
private var connection: NWConnection?
static let shared = DataSender()
private init() {}
func configure(host: String = "localhost", port: String) {
if self.host != nil {
stop()
}
self.host = NWEndpoint.Host(host)
self.port = NWEndpoint.Port(port)
}
func start() {
guard let host, let port else {
NSLog("Host and Port must not be empty")
return
}
connection = NWConnection(host: host, port: port, using: .tcp)
connection?.stateUpdateHandler = { newState in
switch newState {
case .ready:
NSLog("Client connection ready")
case .failed(let error):
NSLog("Client connection failed: \(error)")
self.connection?.cancel()
case .cancelled:
NSLog("Client connection cancelled")
default:
break
}
}
connection?.start(queue: .global())
}
private func sendData(_ message: String) {
guard let connection = connection else { return }
let data = message.data(using: .utf8) ?? Data()
connection.send(content: data, completion: .contentProcessed { error in
guard let error else {
NSLog("Data sent successfully")
return
}
connection.cancel()
})
}
func stop() {
connection?.cancel()
}
}
Next, we need to add the creation of a connection when our application is in the UITest or PerfTest launch mode. For example: you can bind the start to what Active Compilation Conditions are located in our config.
import UIKit
@main
final class AppDelegate: UIResponder, UIApplicationDelegate {
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
#if PERFTESTS
DataSender.shared.configure(port: "8080")
DataSender.shared.start()
#endif
return true
}
}
After setting up the main target, it's time to start writing code inside our perf tests (you can also call it Runner). Next, we raise a Tcp server that will track messages that we receive from the main application.
import Network
/// Сервер внутри нашего Runner-а
final class TcpServer {
private var listener: NWListener?
private let queue = DispatchQueue(label: "TcpServerQueue")
var onReceive: ((String) -> Void)?
func start() {
do {
listener = try NWListener(using: .tcp, on: 8080)
listener?.stateUpdateHandler = { newState in
NSLog("Server state: \(newState)")
}
listener?.newConnectionHandler = { newConnection in
newConnection.stateUpdateHandler = { state in
switch state {
case .ready:
self.receiveData(connection: newConnection)
case .failed(let error):
NSLog("Connection failed: \(error)")
default:
break
}
}
newConnection.start(queue: self.queue)
}
listener?.start(queue: queue)
} catch {
NSLog("Failed to start listener: \(error)")
}
}
private func receiveData(connection: NWConnection) {
connection.receive(minimumIncompleteLength: 1, maximumLength: 4096) { [weak self] data, _, isComplete, error in
guard let self = self else { return }
if let data = data, !data.isEmpty, isComplete, let receivedString = String(data: data, encoding: .utf8) {
onReceive?(receivedString)
}
if error == nil && !isComplete {
self.receiveData(connection: connection)
} else {
connection.cancel()
}
}
}
func stop() {
listener?.cancel()
listener = nil
}
}
Recording data
After we have collected all the data in our test, we need to save all this data. The easiest way is to use XCResult. We may also want to write some additional data as an attachment. And to our aid comes XCAttachmentwhich allows you to add all the information that corresponds to Data or, for example, XML.
let savedData = prepareData(from: result)
let attachment = XCTAttachment(data: savedData)
add(attachment)
Follow-up analysis
After we have collected our results and saved them for further analysis, we can build a histogram based on our measurements that we have created and clearly determine the upward or downward trend in numbers. For example, if we are testing a hypothesis about speed degradation, we run two runs of performance tests – before making changes. The second – after applying changes with different analysis. Then we compare two values using, for example, diff and determine the trend line.
Naturally, one can write fairly simple scripts to compare values and from this determine the trend line.
Conclusion
Perf tests occupy a special place in the overall testing pyramid, and if you are just starting to develop your application, they may be redundant. However, if you have already passed the initial stage of the application development and are ready to spend resources on performance, protecting yourself from unconscious changes and feedback from real users, then this type of testing can be useful. Each team must decide whether this is appropriate. But it is worth it.
