Testing metrics that you definitely need to implement in the process

Let’s talk about testing metrics: which ones will be most effective for implementing into your testing process, how they are taken and calculated. Using examples, I will show how this process is organized in Innotech.

Why you need to take metrics

Testing metrics are used to track efforts to ensure the quality of released code. With their help, it is possible in numerical terms to get an idea of ​​​​achieving a given level of quality or goals. The visual presentation of the results forms a visual picture of the testing process, which can show bottlenecks.

The following metrics are used in the testing process:

• to track the progress of the team in terms of the project, deadlines and other time periods;

• qualitative assessment of the current state of the system;

• quality control of the testing process;

• setting goals and effective planning based on an understanding of existing problems.

We should not dwell on the current quality of the functioning of the system and, more importantly, the processes. It is these characteristics that are the basis for the growth of the team’s efficiency and performance indicators. The rationality of the use of the human resource is directly tied to the overall performance. It’s a shame to have a team specialist whose potential is not realized even by 50%. For such inattention, it is quite possible to earn not the most pleasant bonuses, up to the loss of reputation and money.

What metrics are most often mentioned in testing articles

When testing on projects, a number of metrics can be identified that are most often mentioned in most training courses and articles.

Passed/Failed Test Cases. It is used to estimate the ratio of successfully passed tests to those that ended with errors. The metric helps to evaluate the success of passing tests.

Not Run Test Cases. Shows the number of tests to run for this project. The metric helps you identify the reasons for failing tests and how to fix them.

Open/Closed Bugs. It is formed from the ratio of open bugs to closed ones. The metric evaluates the speed of fixing bugs, and also allows you to identify the reasons why bugs remained unclosed.

Reopened/Closed Bugs. Calculates the ratio of reopened bugs to closed bugs. The metric shows the effectiveness of closing a bug to developers and will help to identify the reasons why bug fixes are at a low level.

Bugs by Severity/Priority. Total number of bugs by severity/priority. The metric shows the quality of the code provided for testing.

What metrics need to be taken additionally?

In addition to standard metrics, Innotech also uses a number of additional ones. They allow you to get an objective picture of the process.

Coverage with test scripts and checklists

Applies to requirements, user stories, acceptance criteria, risks, or code. Metric-based testers can focus on identifying features not covered by test documentation. Uncovered functionality carries a certain risk, which will be unacceptable on many projects.

You can use the requirements trace matrix to evaluate coverage.

Percentage of written test scripts and checklists

As a basis, you can take the total number of features, the trace matrix, user story. The coverage density of the requirements should also be evaluated. To do this, it is worth using atomicity – each functionality should be divided into the most atomic features, which should also be covered by test documentation.

Percentage of completed work on preparing the test environment, test data

Testing is performed on test data, so it is important to understand how complete and correct they are. Checks must be accompanied by the necessary test data covering all aspects of the code being checked. To prepare them, we use tables of domain analysis and pairwise testing, as well as tables and diagrams of transitions and states.

The metric is necessary to assess the readiness of the testing start criterion. It is necessary to start from the amount of test data required and track progress by the ratio of readiness to the required norm.

Test execution metrics

It reflects the number of passed test cases and failed cases, the ratio of completed cases to the total number of cases and failed cases to the total number of cases, the average time to complete a case. We monitor this metric in order to correctly distribute the resources of testers, prioritize cases, involve additional forces in the passage of regression, and determine the current dynamics to adjust the timing.

As a rule, this is implemented by RMS systems; you can also track the number of completed cases in excel tables.

Defect Metrics

This includes measures of density, number of defects found and fixed, failure rate, and results of confirmatory tests. This allows you to get information about the quality of the product in a specific period of time as objectively as possible. Of course, usability is difficult to assess based on the density of defects. However, the status of the product in relation to requirements is quite good. Also, relying on information about blocking and high-priority defects, you can adjust the development orientation.

It is necessary to collect this metric in terms of regressions and the current state of the project. You can track by failed test cases, or keep additional statistics in excel tables.

Information about the status of tasks, specialists, workload and labor costs

The metric tracks the workload of the testing team, the distribution of labor costs among tasks, and so on. The hardest thing to implement a metric is if the project does not support daily work logging. However, the positive impact of the metric will allow you to correctly distribute time among tasks, track the actual and expected assessment of tasks, and adjust the assessment in the future.

You can track metrics using daily work logging in Jira, or keep records in other systems that are used on the project.

financial information

The metric includes the cost of testing, the project, and other team costs.

The metric is necessary for planning financial resources, as well as to track the “leaks” of these resources. This metric is necessary because it allows you to avoid holes in the budget and more effectively show the costs of the team to stakeholders.

It is implemented by taking into account the financial expenses of the team, as well as planning future expenses.

Conclusion

Monitoring of processes should be carried out regularly, comparing the results at different stages of development. The relevance of the metrics is controlled by the specialists implementing them, and their results should be analyzed by the stakeholders to correct the correct path.

You can write an entire article about how to shoot metrics. To save time, here’s a summary:

• in order to get the opportunity to win big in the future, it is necessary to implement the removal of metrics into processes now, sacrificing only a small amount of time;

• an integral component of the quality of these metrics will be the control of specialists.