Apple WWDC 2020: what’s new in iOS testing
But first, a brief historical excursion: Apple WWDC (WorldWide Developers Conference), or just dub dub, is a confe that Apple has been holding in California since the late eighties. This year the conference was first held in an online format. And if before the tickets were raffled, and those who did not receive the desired email had to be content with a video from the site https://developer.apple.com/videos/, then for obvious reasons this year there were no other options: everyone watched the video.
So, what could be spotted on testing there?
I must say right away that at WWDC 2020 there was no big general session devoted to testing in the Apple ecosystem, as in previous years (Testing in Xcode 2019 and What’s new in testing 2018, 2017) Testing novelties in 2020 were smeared into six mini-sessions. Go!
XCTSkip for your tests
Xcode 11.4 added a new API to control the launch of tests depending on conditions – XCTSkip.
Often in tests, especially integration tests, there are conditions or requirements that are not easy to ignore. For example, the application has some specific functionality for the iPad, which does not work on the iPhone. Or some features for a specific version of the operating system.
And before, when the tests reached such cases (checking ipad-only functionality on the iPhone), there was a choice:
- Finish the test suite;
- Mark the test as passed and go further;
- Flake the test.
Now we have an error in which the current test ceases to be executed and is marked as missed.
Thus, now in XCTest there are three statuses for the passed test instead of two:
Interrupt and alert handling in UI tests
Interrupt and alert handling was in XCTest before, however, in the session, the mechanism of its operation was disclosed in more detail. I found the new functionality interesting in Xcode 11.4, iOS / tvOS 13.4, and macOS 10.15.4, namely reset permissions (aka protected resources).
The bottom line is this: if earlier, for example, in test # 1 you gave the application access to the camera or contacts, then later, in test # 2, #n this access is not so easy to select. To do this, you will have to reinstall the application.
Now, using the API for authorization reset for protected resources, you can select previously issued access:
Class XCUIApplication {
open func resetAuthorizationStatus(for: XCUIProtectedResource)
}Resetting permissions makes the application behave as if it had never before requested the user access to protected resources.
This allows you to go all the way with the issuance and collection of permissions for contacts, calendar, photo, microphone, camera and geolocation. On iOS, you can also optionally reset access to Bluetooth and Keyboard network access, and starting with Xcode 12 / iOS 14, to Health data. On Mac OS, you can reset access to the Desktop and Downloads directories.
Below is an example of how to reset application access to photos:
// Example
func testAddingPhotosFirstTime() throws {
let app = XCUIApplication()
app.resetAuthorizationStatus(for: .photos)
app.launch()
// Test code...
}It is important to remember that often (but not always) the application is killed when resetting permissions.
More details here, here and here.
Eliminating animation lags with XCTest
Animation lags, or hitches, are behaviors when a frame appears later than expected.
The lecture tells how to prevent the appearance of animation lags in your application by measuring and testing with Performance XCTests.
The best practices are also given, and it is determined which lags are tolerant and which ones should be paid attention to:
Describes why Critical Logs deserve a thorough investigation and correction. The topic of animation testing itself is quite extensive and worthy of a separate article, so we restrict ourselves to the introductory part and a link to primary source.
Triage and diagnosis of fallen tests
Often repairing dropped tests is a pain that takes a lot of time and resources.
Xcode 12 introduces a new API that should make it easier to fix dropped tests. The API should help answer questions faster: what, how, why, and most importantly – where did it fall?
If earlier after the test fell, you had to look for the place of the fall in
Issue navigator or report navigator, then with Xcode 12 the search process has been simplified: now the crash site is highlighted in the test itself.
An error with highlighting appears if the line refers to some other line in the future:
And in red, if an error occurred directly in this line:
A convenient new feature is opening the code editor not in a separate window, but directly in the report navigator:
In addition, a new XCTIssue object has been added to Xcode 12, which, in addition to encapsulating error data that XCTest previously collected (message, path, line number and the “Expected” flag), now adds:
- Distinct types;
- Detailed description;
- Associated error;
- Attachments
Write tests so they fall
The goal of testers is to write tests to see them green, passed, because this means that the product can be shipped to end users. Nevertheless, writing tests to see them as flawed is also necessary, because a flavored test is very likely to be a found bug. Thus, you need to write tests with an eye to the fact that if they fall, we would have enough information to investigate.
So what is being offered:
Use human-readable messages in assertions:
Make sure to use the type of assert appropriate for your situation:
Unwrap optional so that your tests fail, throwing an error, and not crash. Swift provides several ways to do this, but tests typically use XCTUnwrap, which is a simplification of the guard let construct.
Use waitForExistence () instead of sleep () for asynchronous wait.
Use XCTContext.runActivity () to increase readability of the test execution log:
And if you want to add additional logging, you can add an attachment, attach a screenshot or debugger output, as here. This feature is especially useful if your tests run on a CI / CD.
More details here.
Get Test Wound Results Faster
It’s a shame when on Monday morning you find that a long job launched on Friday night did not work until the end, hovering in the middle or even at the very beginning. And you have to start the working week with a debriefing: why did this happen? How to avoid a similar situation in the future? How could I get nine thousand for cocktails in one evening?
Xcode 12 introduces anti-freeze tools. This is a new test plan option called Execution Time Allowance.
When this option is enabled, Xcode sets a time limit for the execution of each test.
If the limit is exceeded, Xcode does the following:
- Collects a report (spindump);
- Kills a frozen test;
- Restarts the runner test so that the rest of the suite can execute.
The report (spindump) shows which thread, which function spent the most time. This will allow you to see with your eyes the bottleneck of your tests before your morning coffee / tea cools.
By default, 10 minutes are allocated for each test, and if the test is completed faster, then the timer is reset to the next test. If you need more / less time for each test of your test suite, you can change the default value in the test plan settings.
You can also do this with the xcodebuild command option:
xcodebuild option
-default-test-execution-time-allowance Similarly, you can set the maximum test execution time:
xcodebuild option
-maximun-test-execution-time-allowance Even if you need to set the runtime for a particular test or test class, this is also possible using the executionTimeAllowance API:
Class XCTestCase: XCTest {
var executionTimeAllowance: TimeInterval // с округлением до минуты
}Fine tuning the execution of a particular test will save you time, but this is not all that can be done to speed up the passage of a long test suite.
Xcode 12 allows you to run tests on multiple devices at the same time. This feature was called Parallel Distributed Testing. The benefits of running tests on several devices are obvious – a decent time saver.
But, unfortunately, there are pitfalls: the order of running tests in parallel is not deterministic, there is no guarantee that on device # 1 after test number 5, test number 6 will be performed. This fact must be taken into account when planning to run tests using Parallel Distributed Testing.
In general, the idea of running tests in parallel is not new. Such an opportunity existed before Xcode 12, but it was in Xcode 12 that it became possible to run tests on real devices (so far only using xcodebuild).
The command to run parallel distributed tests is as follows:
xcodebuild test
-project MyProject.xcodeproj
-scheme MyProject
-parallel-testing-enabled YES
-parallelize-test-among-desinations
-destination 'platform=iOS,name=iPhone 11'
-destination 'platform=iOS,name=iPad pro' More details here.
This concludes the review of new test features with WWDC 2020. Thank you for reading to the end.
I hope you find this article useful. Happy testing!
