Where are we and where are we going? Making the testing process transparent with Klaros TMS

It is important for the test manager to have up-to-date information about the test cases used, the time spent on their execution, retrospective statistics on the number and success of manual tests (and, ideally, also automatically extract the results of automated tests in CI / CD), and also other documentation about the testing process and its results, and this need was implemented in test management systems (Test Management System, hereinafter TMS). There are a large number of commercial TMS solutions on the market (such as test rail, PracticeTest, Zephyr Squad for Jira, XQual, Qase, testiny), which sometimes also cover requirements management, release management, and KPI compliance. In this article, we’ll cover the basics of using Klaros TMS (which can be used for free in the Community version) and talk about the Local TMS approach offered by Jetbrains.

The main task of the test management system is to prepare a formal description of test cases, determine dependencies between test cases, create test suites for various types of testing (for example, regression or smoke tests), control the execution of test code (or edit the results when performing manual testing) and accumulation of test results. TMS typically supports multi-project management and provides a set of command line tools and/or APIs for reporting and managing test case results. Additionally, TMS can link test cases to the release schedule, generate reports on the results of test execution (for example, graphs of the percentage of success over time or a list of detected defects), in some cases, task lists and release documentation management are also supported (creating a test report, logging error messages from users).

A significant part of TMS are commercial or cloud-based subscription solutions, or support free functionality with restrictions (for example, Qase or Testiny). We will consider a non-commercial product Klaros TMS Community Editionwhich can be installed on the organization’s servers and used without restrictions.

Klaros TMS

Functionally, Klaros TMS provides the basic testing process and can be used as an alternative to TestRail. The non-commercial version does not support working with requirements and test coverage, nor can you perform test run scheduling and project management. At the same time, all the necessary functions for managing test cases, executing tests and saving a test report, preparing reports will be available.

To start Klaros TMS we will use Docker image (also available installation files for Windows and linux), for this we will clone the project from Github and build a new image (it will be necessary to install Docker on Linux or docker desktop on Windows / MacOS):

git clone https://github.com/klaros-testmanagement/klaros-docker
cd klaros-docker/ApacheDerby
docker build -t klaros .
docker run -d --name Klaros -p 18080:18080 -v klaros-data:/data klaros

Once launched, the Klaros TMS web application will be available at http://localhost:18080, for the first login you can use the username and password admin/admin.

First of all, let’s start by creating a project:

After creating a project, you can add integrations with task management systems (issue tracker), for example, Jira, Github, Gitlab and others are supported:

Next, select the active project (the round icon to the right of the project name) and now the ability to run tests and receive test reports becomes available. Then we define the system that we will test (in our case it will be a console calculator), Define → Systems Under Test → New.

Next, let’s create a Test Case: Define → Test Cases → New. When defining a test case, you can additionally specify the priority (low – medium – high), status (draft, approved or temporarily skipped), how the test will be performed (Manual / Automatic). Now we will create two tests – an automatic test for adding numbers and a manual test for subtracting, to be able to edit, tests must be created in the “Draft” status, which will need to be replaced with “Approved” before starting the tests

For a manual test, you will also need to fill out a test case card and indicate the preconditions and postconditions, the expected result, a description of the execution steps, and a link to the documentation. Issues (created from the application or linked to external systems) can later be added to the created test.

Test cases can be combined into test suites (Define → Test Suite → New)

Next, let’s try to run a manual test, to do this, go to Execute → Run Test Case and select the “Execute” action. A dialog will be displayed sequentially for each test step with the ability to mark the result of passing the test (success-failure) and, in case of unsuccessful execution, additionally download attachments (logs, screenshots) and a text description of the actual behavior.

As a result of passing the test, an issue can be created (it can also be transferred to an external ticket accounting system).

When running an automated test, it will be necessary to load a machine-readable file with the results of the test execution and clarify which file format was used and in which test environment the test was run. Klaros supports a large number of test frameworks, including *Unit (JUnit XML format), ctest, GoogleTest, JMeter, Fitnesse, Valgrind, and more.

The entire test run history is saved and can be obtained as an HTML/PDF report (Evaluate → Reports) for System Under Test, Test Suite, Test Run History. You can also get statistics on running test cases Evaluate → Test Case Results, running tests, statistics on Test Suite. It also provides general information on the active project in the Evaluate → Dashboard panel.

Jetbrains Local TMS

The Local TMS approach stores plan and test result data directly with application source codes as Markdown files and is supported in Jetbrains IDE using a plugin Test Management (currently in the experimental status, the documentation indicates that the formal specification is not yet ready and file formats may change). To use Local TMS after installing the plugin, you need to enable Markdown support for tests: Preferences → Tools → TMS → Markdown.

The definition of tests starts with the definition of Test Suite (File → New → Test Suite). A name.t.md file is created, in which tags, test cases and test execution steps are defined, for example:

# Calculator
Tags: calculator
Meta: app = calculator

## Проверка калькулятора
* Test subtraction
    * Проверить вычитание 12-2=10
    * Проверить вычитание 18-21=-3
    * Проверить вычитание 24-24=0

To determine the results of test execution, you need to create a file with a report (File → New → Test Case, select the appropriate tests) and add labels to the corresponding test steps [ok] upon successful completion or [fail] when an error occurs.

# test2

* [fail] Test subtraction
    * [ok] Проверить вычитание 12-2=10
    * [ok] Проверить вычитание 18-21=-3
    * [fail] Проверить вычитание 24-24=0

Additional metadata may also be specified,

for example, with @ you can add the tester’s login. To view the history of passing tests, you can use the View → Tool Windows → TMS panel, you can also link test code and test cases from there (New Test for Test Cases in the context menu), after which a test function stub and comments for test steps are created:

class Sub {
    //
    @Test
    fun Test_subtraction() {
        // Проверить вычитание 12-2=10
        assertEquals(10, sub(12,2))
        // Проверить вычитание 18-21=-3
        assertEquals(-3, sub(18,21))
        // Проверить вычитание 24-24=0
        assertEquals(0, sub(24,24))
    }

A test can be linked to the corresponding test case by defining an annotation class and using this annotation, together with the test cases identifier (will be written as a sequence of numbers written in the test cases md definition in ## before the main description, for example ## 1 Test subtraction can be addressed as C1):

annotation class LocalTMS(val value:String)

...
@Test
@LocalTMS("C1")
fun Test_subtraction() {
//...
}

We considered two approaches to creating a TMS: a free version of Klaros TMS (it can be used as a standalone tool for describing test cases and supporting releases when organizing the work of a test team in both automatic and manual testing) and a new approach to integrating test cases and test reports that are organized directly in the source code of the project and use the familiar development and testing practices of versioning and navigating through the source code.

In conclusion, I invite everyone to free lesson from my friends at OTUS on requirements testing methods. In this lesson:

Let’s study specific requirements testing practices.
Consider using User Story and Acceptance Criteria to test business requirements.
Consider how the requirements testing process is built in Agile teams.

You can register using the link.