How to choose mobile devices for testing and not adjust
Hello! My name is Anastasia Sharikova, I head the testing department at Bookmate and lead a telegram channel Yet another QA.
I am sure that many, when they came to the position of QA Lead, they expected complex and interesting tasks in building processes, setting up a system of self-tests, establishing functional and non-functional testing, and, in the end, tasks for managing employees. But in the end it turned out that all this is there, but it is combined with household items such as calculating salaries, determining the budget for the department and other problems associated with finances – and often there is no escape from this, especially in small and medium-sized companies. Nevertheless, attentiveness to such topics is a great way to further improve the quality of your product and find and prevent problems from the very early stages.
Most of the heads of the QA / testing departments, whose employees work with mobile applications, at some point are faced with the task of updating the fleet of mobile devices and deciding what to generally test for employees. And if this is not a question for seasoned specialists at all, then it is often difficult for a newcomer to navigate this issue, and today I will try to talk about what you can rely on in your choice and what difficulties and pitfalls are in purchasing devices for testing.
This article will focus on the selection of devices for manual testing, since the selection of farms for autotests has its own characteristics, and it is better to talk about this topic separately.
Of course, at first glance the topic may not seem so significant – they say, why take a steam bath, you can take the most popular, but more expensive, and test the rest on simulators and emulators. But in reality, a huge number of bugs of all types can be caught only on a real device, and manual testers also usually need to work on something to check daily tasks.
Why is it better not to completely throw off these tasks to other departments, for example, procurement, if any? Because they usually do not understand the features and metrics of your product specifically, and there is a chance to miss something important.
In most cases, our task is to create a universal set of devices that will cover the maximum selection of combinations of axes, manufacturers, models, screens and types of smartphones and tablets, and do it all on our budget.
The main parameters that affect the selection:
- External analytics: here, international collections by the most popular devices (such as, for example, deviceatlas.com) and trivial options like sorting Yandex.market by the most popular devices can come to our aid. It is important here not to fall for the “best” phones, because the opinion of experts often diverges from the opinion of ordinary consumers. The main thing is to remember about which country and Central Asia your product is oriented to, and be based on this condition.
- Internal analytics: in this case, we have a huge number of options that you can rely on, moreover, these are the most clear data – and it is precisely about our product. What can we use here? Data from analytics systems such as Firebase or device information from Google Play. And it is important not to forget to take into account not only the devices and axes that your employees use, but also those on which the system most often catches errors and crashes.
- Market Trends: Every month new players and devices enter the market, axis updates and new market leaders appear, and all this is important to consider. Examples are how much more devices with a “shutter / edge” or US sanctions against Huawei have become over the past year.
Do not forget about the features of our audience:
It is always important to remember the specifics of your product and its target audience and try to avoid subjectivity – even if it seems to you that the iPhone 11 Pro is the crown of Apple’s creation, it won’t bring you much benefit if the audience of your photo application is schoolchildren, which parents gave the sixth or seventh iPhone. The same thing works in the opposite direction – if you know that wealthy people use your service and it will be installed mainly on the latest model iPad Pro – you will have to fork out for it, otherwise wait for trouble.
It is especially important to note about the countries of use – almost all countries have their own “chips”, so the average application audience for conditional India will differ from users from Scandinavia.
Important features and chips on the example of iOS and Android:
Of course, the article does not cover all the difficulties and features, but I will try to mention the main ones:
- Compared to android, there are few devices, and they do not come out so often, similarly with the axes that need to be taken into account.
- It’s easier to keep an adequate averaged set of screen types – and not to forget that some have retina, this may be important for some types of tests. But, of course, not always. However, the main typesetting bugs can be traced to a combination of four to five iPhones from the fifth to the eleventh. And do not forget about the curtain!
- Consider which versions of iOS you support and try to make them evenly distributed across devices.
- Do not forget about the magic of Split View mode and try to keep at least one device that supports it
- The problem with the purchase can always be the price, especially if you try to produce customized versions in the first week of sales – but, on the other hand, iPhones don’t “go bad” for a long time, so this can be considered a long-term investment. Of course, someone here can argue with me, but in my practice it was the Apple technique that lived longer with constant use for testing needs.
- A lot of devices and this is a pain. And here it’s definitely nowhere without analytics – because it is on it that your audience should be evaluated in the case of an android, otherwise you will have to buy anything horrible, and this does not lead to productive results. Again, there is a temptation to test on what is fashionable, modern and fast, but if your users are using Chinese Oppo, then you may not know that there, it turns out, nothing starts up.
- Take into account the peculiarities of the android shells and the store – especially carefully work with those that disable the default Google Play services, including stor. Try to keep different options in the park, including the “bare” android.
- If with tablets in Apple everything is quite simple, then in android it is pure madness, because most of them have old axes, poor quality screens and very slow hardware. Of course, this was to be expected from tablets for 1999 rubles, but if your audience uses them, and you make a heavy toy, then it can be cut down right at the start. Life example: the company I worked for concluded a contract with one of the phone manufacturers that they pre-install our application to them, but when the phones came to the office it turned out that they had so little memory that even pre-installed applications did not start from the box. Of course, I had to simplify our service.
- Remember that despite the traditional attitude towards android as cheap phones, in recent years their flagships have been comparable to Apple’s flagships, and even more, besides they often have such experimental features as beveled screens, clamshells and more.
Where to buy?
I hope that the reader of this article is lucky or lucky, and this question will not concern him – he will only put the task in Jira for purchase and in a couple of days will receive a fresh phone. But what if you work in a company where you need to make this decision yourself? There are several options:
- Official representatives: Yes, often it can be more expensive, but most likely it is more reliable and better guarantee. And a life hack for those who have read up to this part of the article: not always, but often, if you write a letter to the representative office of a particular manufacturer about how much you need a certain device, explaining how you will use it, you can be sent it either for free and forever , or for temporary use. And this is often very relevant.
- Retail Stores: here I combine all the stores that do not sell leftist, with checks and a guarantee, for example, my own. Gray is not always bad, especially for iPhones, for example. The main thing is to buy in verified places, and not in a tent in the market.
- Avito and others: Of course, here I am combining all the options for buying b / y from private traders. Of course, this should be avoided, but if you realize that you need a nosebleed phone that has not been released for 3 years, and your users use it stubbornly, then it’s sometimes worth taking the risk.
- Staff: and another life hack – if you ask your colleagues, it will surely turn out that a couple of employees have an unnecessary phone in their nightstand, which they can nobly give to your department for temporary or unlimited use!
An important financial point – it is often better to take a device with the expectation of a long term of use and simple repair, more expensive, but to last for a long time, and not cheaper, but you will have to change them every three months. But sometimes – on the contrary, it makes sense to buy something bad, because, again, in my experience the most rabid bugs are caught on the most nightmarish phones and tablets.
Learn more about the OTUS QA Lead course.