Testing digital TV receivers: how to transfer a test model from TestRail to a new instrument

My name is Alexander Ivanov, I am testing digital television receivers in GS Labs…

In this article I will tell you about how our team gained courage and replaced the inconvenient and expensive TestRail with a new tool test management. The article contains a description of the stages of moving tests and autotests, setting up a new tool, as well as a direct description of the testing processes for TV receivers and their software.

About me: during his time at the company, he went from a tester to a lead testing engineer. From time to time I give reports at conferences for testers.

GS Labs is a developer of integrated solutions for the formation of ecosystems for the creation and delivery of digital products based on proprietary technologies. More than 12 million subscribers of telecom operators in Russia, the CIS, East and Southeast Asia use GS Labs products.

Test object

Television appeared at the beginning of the 20th century and has been developing continuously since then. With the transition from analogue to digital broadcasting, television has opened up new horizons for itself. The development of microprocessor technology has made it possible to create software and hardware solutions for receiving and viewing digital TV content.

According to the content transmission medium, digital TV can be conditionally divided into:

• Essential,

• Cable,

• Satellite,

• IP television.

A digital television receiver (set-top-box, STB) can be used to receive, as well as for further decryption of content. The receiver uses its own software to process digital content. The set-top box itself, as well as its software, are the object of development and testing at GS Labs.

Testing set-top boxes

As already mentioned, a modern TV receiver has its own software. The main task of the receiver software is to decrypt encrypted content. For successful decryption, the subscriber must purchase from the operator a subscription to the corresponding package of TV channels.

In addition to the content itself, the operator broadcasts advertisements from third-party clients, which must also be displayed correctly by the receiver. There is also additional information support for subscribers, which from time to time informs subscribers about preventive maintenance at the head-end station, receiver updates and other information depending on the needs of the operator.

Like any software, the receiver software also needs to be tested.

Test bench description

A stand identical to that of the operator is used for testing. Test benches include the most common head equipment in Russia: Harmonic, Teleste Luminato, WISI, Ericsson, PBI.

The main element of the stand is multiplexer (hereinafter MUX). The MUX is a combined digital device that allows multiple input signals to be transmitted alternately to one output. It is needed in order to create “channels of your choice”: add audio tracks, teletext, advertisements to the original content, and also encrypt it using the Conditional Access System (CAS), which is also installed on MUX. Both the COURT server and other auxiliary servers are usually virtual machines.

Test data can be roughly divided into:

• configurations (live emulation). It is an emulation of TV broadcast;

• streams Recorded TV broadcast.

Testing is done on both threads and configuration. The main plus of the configuration is that we always deal with the latest version of servers.however, they need to be updated regularly. The main advantage of streams is ease of use, no server configuration is required, everything is already configured and can be reused an infinite number of times. However, in this case it is not possible to use the latest version of the transmitting part.

An approximate diagram of the test bench is shown in Figure 4. During manual testing, the receiver is connected to a signal source (live emulation or stream), the image is transmitted from the receiver to the TV, for example, via an HDMI cable. Using the remote control, we look for a signal, switch channels and check the functionality of the set-top box software.

The test case will depend on the functionality we are testing. Different commands are responsible for different receiver functions. For example, if this is a COURT testing, then the case carried out on the stream will look like the table below.

Test automation

As in any software testing, we also have the task of automating our test cases.

Most of the pros and cons of automation are the same as for any automation, be it client-server or mobile applications.

Pros:

● speed

● ensuring repeatability

● reduction of labor costs for testing

Minuses:

● cost

● “pesticide” effect

● the need for regular support of autotests

Our autotests are written in Python 3 using the Behave library. For automated interaction with the receiver, an API is used (instead of a remote control during manual testing), which is regularly updated and improved.

For storing tests, many options have been tried all the time: Excel, Test Log, TestRail. The most successful system at that time was TestRail.

During autotests, their results were transferred to TMS through the autofunction_name field, or rather through a numeric prefix at the beginning of the name (a legacy of older TMS systems). It was decided to abandon the use of the name field, since the name was understood to be subject to change.

In total, we have the following fields (for example, one of the autotests):

name	Broadcast infocas
autofunction_name	01_01_Broadcast infocas
automated	yes

What the corresponding autotest looks like:

@short_term 
Scenario: 01_01_Broadcast infocas
Given we switch to channel "12" on "server"
When we send "broadcast" infocas message "Broadcast Test Infocas" for test "01_01"
Then we "see" infocas message displayed for test "01_01"

As you can see from the example test, it was written using BDD. Each of the lines written in a “natural” language means some code written in a programming language, and the values ​​in quotation marks are passed as parameters. Thus, to write autotests, knowledge of programming languages ​​is not required, the main thing is to know the tested functionality. In general, I met an ambiguous attitude towards the use of BDD: the approach has supporters and opponents. I do not classify myself as either one or the other, but I note that in our case the approach justified itself. Manual testers take an active part in writing autotests, because There are a lot of tested functions and it is almost impossible for the automation team to understand the nuances of each of them.

At the end of testing, a report was generated using TMS tools.

Tool change for testing

To change the usual TestRail by switching to a different test management system, we were prompted by a simple human “expensive”. In this regard, we conducted a review of the systems existing at the time of 2019. Based on the combination of several factors, we chose a new Russian development Test IT. Having started using the new TMS, we discovered its other advantages, for example, the automatic generation of reports in dashboards.

The move itself took about a month. In 2021, everything is already simple in Test IT: import the test model into .xls, correlate any necessary fields and test for health, but at the end of 2019 this functionality was not yet implemented.

In several stages we unloaded the tests in * .xml and successfully imported them into Test IT… And then the question arose of how to transfer the results of autotests – during manual testing, such questions did not arise, rather, there were questions how to generate a report and which fields to include.

As it turned out later, the field autofunction namethat we used cannot be ported to Test IT to indicate results, but the autotest uuid can be used. We found a solution: knowing the uuid of the autotest, you can get its name and use it to transfer the results. The numeric prefix has moved to the name of the test case. Further, an autotest with the same name was linked to each automated script. As a result, the autotests themselves remained unchanged, with the exception of the auxiliary module for transferring results to TMS Test IT.

As a result, we have fields:

name	01_01_Broadcast infocas

Today, all new cases are written in TMS itself. Here we managed to use the functionality of common steps in practice and the ability to indicate the result of each of them in the test case. This made it possible to speed up the writing of cases and make it easier for new employees to work with them.

Now it is possible to track the history of passing test cases, on the basis of this, you can detect unstable autotests and have an idea of ​​possible bugs in the software of digital TV receivers.

Upon completion of testing, a report is generated that is automatically generated in Test IT, a fragment of which can be seen in Fig. 5.

Conclusion

Changing the TMS system always requires a certain amount of labor. It is very important that the process is as easy and fast as possible, and does not critically affect the speed of release and the quality of the products.

Even before the start of the move to the new system, the development team held several training presentations in our office, at which we asked questions of interest in advance. I believe that such presentations helped the process of moving to the new TMS.

Undoubtedly, we ourselves also had to figure it out, study the API, and, if necessary, write to support. The main result is that we managed to move to the new TMS without any problems. As for improving the current processes, it will take time to fully understand this. But it can already be noted: the new tool helps to create test cases faster and more conveniently and allows you to visually track the history of their execution.