Quality dashboards at Ericsson, Volvo, Saab

Translation of part of the chapter “Dashboard for Continuous Monitoring of Quality” / “Dashboard for Continuous Monitoring of Quality” of the book “Relating System Quality and Software Architecture” / “Relationship between system quality and software architecture”, Mistrik, 2014

———-

In this section, we present how the three case companies used the concept of release readiness in their dashboards for teams and development projects. The approach of the three companies is different and is based on the history of using indicators in these companies, so not all elements of a successful dashboard are present in all companies. Table 1 provides a summary of each company

The companies operate in three different areas, with similarities (e.g., embedded software development) and differences (e.g., development methodology, vendor dependency). The breadth of companies covered and the commonalities presented on the dashboard show that similar challenges and elements of successful dashboards apply to each area.​

Ericsson AB

Ericsson AB develops large software products for mobile telecommunications networks. At the time of the study, the organization had several hundred engineers, and the number of projects reached several hundred. Projects were increasingly carried out according to the principles of Agile software development and a Lean production system called Streamline development at Ericsson (Tomaszewski et al., 2007). In this environment, different teams were responsible for larger parts of the process compared to traditional processes: development teams (cross-functional teams responsible for the full cycle of analysis, design, implementation and testing of specific product features), network validation and integration testing, and other.

The organization used several measurement systems: to monitor the software development project (at the project level), to monitor the quality of products in operation (at the product level), and to monitor the health of the organization at the top level. All measurement systems were developed using the methods described in Staron et al. (2008, 2010a), with particular emphasis on the measurement system design and deployment models presented in Staron and Meding (2009b) and Meding and Staron (2009).

The needs of the organization have changed from calculating and presenting metrics (about 7 years prior to writing this chapter) to using forecasts, simulations, early warning systems and processing large amounts of data to manage the organization at different levels and provide information from both projects and the line. These needs have been met through research projects carried out within the organization since 2006.

Volvo Car Corporation

VCC is a Swedish automotive and automotive original equipment manufacturer (OEM) based in Gothenburg. VCC developed software and hardware in a distributed software development environment. For a number of electronic control units (ECUs), the software was developed by in-house software development teams who were typically also responsible for integrating the software with vendor-developed hardware. However, the bulk of embedded software development came from external vendors who designed, implemented, and tested functionality based on specifications from VCC (Eklund et al., 2012; McGee et al., 2010).

The resource footprint of an entire automotive project was significantly greater than that of telecommunications projects due to the fact that both OEMs and suppliers (tier 1 and tier 2) were involved, and automobile development projects were typically conducted using a product line-based approach. reference architectures (Gustavsson and Eklund, 2011). However, we studied one team that was comparable in size to the teams at Ericsson and Saab EDS.

The organization studied at VCC was a software development team responsible for the software for the climate control system ECU. The team received a set of measurements to visualize development progress and communicate this information to management.

Saab Electronic Defense Systems

Saab EDS develops embedded software and graphical user interfaces for ground-based radar systems. The particular product we were working on was part of a larger product developed by several hundred developers, designers, testers, analysts, and other specialists. The historical project to develop this product was carried out in stages and did not use cross-functional teams. Project management did some manual metrics on problem reports.

Since then, the organization has moved to more Agile processes and cross-functional teams. Significant improvements and optimizations have also been made regarding software build and delivery times. To improve customer value, market competitiveness and profits, Saab AB Electronic Defense Systems in Gothenburg is undergoing a Lean transformation.

Saab EDS has a history of using quality measurement and communication through dashboards. The dashboard presented in this chapter shows how an organization uses a single metric—the number of defects—at varying granularities to provide information about the status of software development.

Ericsson dashboard

Ericsson chooses a dashboard that shows product readiness for release in a compact form. The product release readiness indicator is designed to predict when the product under development will reach the appropriate quality for release (Staron et al., 2012). Quality is measured by the number of open defect reports for a product, which means the correct quality for a product release is 0 defects. Defects can be related to both the functionality of the software and its non-functional properties, in particular performance, reliability and availability. These properties are tested through regularly executed test suites.

The criterion of 0 defects is sufficient only if the other criterion is met – all functionality is tested and all test cases are passed. The stakeholder for this indicator is the project manager, and the software development program is the group that must communicate information to higher management teams in the program and the organization. The indicator (RR, ready for release) has the following form:

where #defects is the number of open defects for the product, defect_removal_rate is the average number of defects removed over the last 4 weeks, test_execution_rate is the average number of test cases passed over the last 4 weeks, and test_pass_rate is the average number of test cases passed over the last 4 weeks. The 4-week period was chosen based on statistical and empirical analyses. These analyzes showed that, based on the length of test cycles and defect resolution activities, a period of 4 weeks was the most appropriate length for this prediction and produced the most accurate results (Staron et al., 2012).

The formula is based on a number of basic measurements (eg tests passed in a week) and derived measurements (eg test pass rate in the last 4 weeks) that show how standardization according to ISO/IEC 15939 is implemented in practice. Each of the measurements in the formula is defined according to the standard with its measurement method and measurement function. The stakeholder has the means and capacity to respond and get the project back on track if necessary.

Figure 8.3 shows how the indicator is distributed throughout the organization on a daily basis – in the form of an MS Vista Sidebar gadget (an example of a compressed visualization). The gadget is a dashboard with an indicator. It is supplemented with an Excel file with trends for the dimensions in the RR formula.

The gadget content shows that the project will need 2 weeks to reach release quality (weeks before release). The presentation of information is simple and concise, providing the stakeholder (the manager of the product development project under study) with the information they need. The gadget also contains an information quality indicator (information quality assurance), which is abbreviated as IQ (Staron and Meding, 2009a). Details about the base and derived dimensions are available in the associated MS Excel file, which opens when you click on the gadget.

In addition to the gadget, the team has a supporting measurement system that tracks dependencies between architectural components, both explicit and implicit. The measurement system is based on monitoring how software components change over time and which components change together (Staron et al., 2013), with an example in Figure 8.4.

Dependencies are visualized very concisely and allow teams to quickly find implicit dependencies that can lead to architecture erosion. New dependencies require updating test strategies and therefore impact the release readiness indicator (new test cases to execute, requiring more time to test the product before release).

Dashboard in VCC

In the case of VCC, we studied how one software development team could track development progress using a dashboard consisting of three components—requirements management, model version control, and testing progress. The team's interest was to report on the software development status of a single ECU for a family of modern Volvo branded vehicles. The dashboard was designed and developed in collaboration with a stakeholder from the team who had insight into the company's development process and practices. A stakeholder was nominated by the team to represent the team's overall opinion. A dashboard for monitoring development progress for a team is presented in Figure 8.5.

This dashboard presents indicators for monitoring trends in software development by tracking the pace of ECU functionality modeling (table under the heading “Indicators”). The team chose four indicators to monitor commit rates and trends:

• Commit trend: the main indicator reflecting the readiness of functionality, defined as the difference between the moving average and the number of checks in the last week. If the number of tests decreases relative to the moving average, this indicates a high probability that the team is moving towards the testing phase. In a concise form, it reflects the team's readiness to implement the functionality.

• Number of commits last week: an indicator that records the “temperature” of functionality development.

• Number of commits this week: An indicator that tracks activity in the current week to help the team check how status is developing over time.

• Commit rate: indicator showing the average level (moving average) of checks.

• Heat map of revisions by model per week: An indicator that visualizes how stable the software architecture is. This is part of the internal quality measurement for the team and helps identify areas where the architecture needs to be improved.

Since the team works according to the Agile methodology, where functional testing is an integral part of model development, the number of commits can be related to both the addition of new functionality and changes to existing functionality caused by fixing defects. However, regression testing was not included and therefore additional measurement was required to monitor the progress of the core quality assurance of the developed functionality.

The team monitored three supporting trends on the dashboard to ensure that the indicators' assumptions were met:

• Identify and breakdown requirements (number of REQPRODs): To assess release readiness, the team checks for new requirements in the backlog. New requirements indicate that new work remains to be done (such as new model updates or new model development).

• Simulation (number of new checked files): to determine the readiness of the team with the product, where it is necessary to evaluate the development trend. A heat map of model revisions shows the internal stability of the architecture.

• Test progress: Number of regression testing tests passed.

The dashboard presented in Figure 8.5 is built on the same principles as the gadget presented in the “Ericsson Dashboard” section, but the information is presented in the form of a stream – three charts in one row instead of one indicator. This more comprehensive view of release readiness was because the team wanted to gain more visibility into the development process and visually communicate status to stakeholders outside the team, such as subproject managers.

The visual presentation of information as a stream is also designed to focus the team's attention on early warning. By visualizing the trend in the number of new/changed requirements, the team can track whether there is still functionality left to implement and test.

When the growth trend of requirements remains stable, the trend of reviews is decreasing, and the number of passed regression tests is stable or increasing, this indicates that the team is close to release (emphasis on decision making).

Dashboard in Saab EDS

At Saab EDS, we studied two dashboards – one for monitoring the internal quality of a software product being developed (Figure 8.6) and one for monitoring external quality (Figure 8.7). These dashboards are complemented by the heatsinks of the assembly. Assembly radiators are used to display the current assembly status of products in the studied Saab EDS organization. If a product appears red (indicating a failed assembly) on the heatsink, people react to this and take the necessary action to make the assembly work again.

The dashboard in Figure 8.6 presents indicators for monitoring the current status and trends of internal software quality (ISO, 2005b), in particular product complexity. This dashboard is used by developers every day because it visualizes data that is immediately relevant to them. Data for indicators is generated by a static code analysis tool and shows the status of quality indicators such as source code complexity.

Early warning of problems is provided by other views in the same dashboard, showing trends for the following metrics (as an example):

• Treemap/Heatmap (center of Figure 8.6 with red intense color indicating problem area and green color indicating high quality area) is used to identify a software module with low compliance. The size of the rectangle is determined by the number of lines of code. This information is used as input to decide whether to clean up or rewrite a particular module.

• The top right portion of the dashboard (Figure 8.6) shows hot spots caused by critical violations, helping to quickly identify high-risk code and easy solutions. Hot spots are the most important parts of software systems that violate the rules set by architects, such as non-compliance with architectural guidelines and interface visibility defects.

• The box in the center right lists the most broken rules by severity. The shaded box represents the number of cases, and the distribution diagram is on the left. This information is used to identify specific problem areas that need to be addressed (focused code cleanup, training, etc.).

• The lower right box in Figure 8.6 shows the complexity of the code. Given that a module with high complexity may be more error prone and more difficult to maintain, higher values ​​require action to be taken on that module.

The dashboard presents multiple aspects of internal quality in a concise and clickable manner, allowing designers to drill down into detailed measures or into the same measures at lower levels of abstraction (such as functions or parts of code). Internal quality includes most of the attributes from ISO 25000 (attributes from the ISO approved parts of the standard). Attributes are calculated at multiple levels of the architecture, from the product level to the module or function level (if applicable).

The dashboard in Figure 8.7 presents indicators for monitoring problem reports, that is, the external quality of the software product being developed. The information is used to quickly identify bottlenecks in the problem reporting process. The small graphs represent the number of problem reports in a given state at a given point in time. The summary chart contains the sum of small charts at a given time. The colored numbers below the summary indicate the delta of problem reports over a 24-hour period.

An upward trend in resolved issue reports, for example, can indicate both positive effects (increased product quality) and negative effects (imbalance between testing and development resources for defect resolution).

———-

You can read about my experience using a quality dashboard, not based on these examples, here – I also use the ISO 25000 series of standards and visualization for decision making

I will be glad to share my experience on the topic of IT product quality, QACoE (QA Center of Excellence), TCoE (Testing CoE) – write to my LinkedIn.