We use them incorrectly.
Author of the article: Dmitry Kurdyumov
Participated in Agile transformations in the largest companies in Russia (Alfa Bank, MTS, X5 retail group), with international experience in a startup abroad.
DORA 4 metrics are taken from the book «Accelerate»a popular book for Engineering Leaders.
DORA includes 4 main metrics:
Deployment Frequency
Cycle Time, sometimes called Lead Time for Changes
Change Failure Rate
Mean Time to Restore (MTTR), sometimes referred to as Time to Restore Service (TRS)
In their words:
«Deployment Frequency and Change Completion Time measure the speed of work, while Failure Rate and Service Recovery Time measure stability. And by measuring these values and continually working to improve them, a team can achieve significantly better business results.»
We are using DORA metrics incorrectly
Measuring DORA metrics alone does not lead to improvement or better business results. There are three reasons for this.
Businesses do not know how to translate DORA metrics into a language that is understandable and meaningful to them.
For example, Cycle Time and Deployment Frequency show how quickly small pieces of work move through stages and into production. However, this does not help the business understand how quickly and frequently we deliver complete features that have the potential to deliver value. Furthermore, it does not matter how quickly you release work if you do not build what the business and customers need.
Next, we will analyze which metrics demonstrate the results of the work of engineering teams in a way that is understandable to the business.
DORA metrics are lagging indicators of speed and stability.
Lagging indicators are those that you can't immediately influence. To really improve them, you need to measure and improve an additional set of leading indicators, such as pull request size, code review time, and code churn.
Continuous improvement in engineering organizations does not happen without changes coming from the developers themselves.
We know that creating dashboards purely for managers and not involving development in the change management process will not produce real changes.
To connect DORA metrics to business results, we need to stop looking at them as an end goal and look at the bigger picture of engineering process improvement.
How do engineering teams create value?
Some people believe that the success of engineering teams should be measured by business results, such as revenue. It is a good idea to align the entire team, both engineering and business, with a single goal. But these metrics are not enough to truly measure development success.
There are two metrics that can be monitored and measured in engineering that are proxies for these business outcomes.
Value Variable 1: Deliver New Features Fast
Any top manager or customer wants engineers to release new features faster. New features help the product grow, make sales, and maintain customer loyalty – both of these processes translate into revenue.
More specifically, I expect the following from engineering teams:
Can we deliver more value with what we already have, with the resources we have?
Can we significantly increase value delivery by investing in new engineers?
Are we delivering what the business has requested on time?
The first two questions are about speed and predictability. The third is about alignment with business goals.
Value Variable 2: Keeping Promises
A very important, yet underrated way that engineering organizations deliver value is by delivering on promises. Think about how often you have to change dates and lower expectations in meetings with the business. If we constantly apologize for not delivering on commitments or have a lot of mistakes and incidents that impact customers, it hurts the business and business loyalty decreases.
Three non-DORA metrics that are great at demonstrating the results of engineering work
Since the goal is to deliver on promises and deliver more features faster, the improvement process must start with our projects. Projects, initiatives, epics, features — whatever they’re called in your organization — are the common language between engineers and the business.
Metric 1: Project Allocation
Metrics «Project distribution» answers the question: «What percentage of our team works on each of our projects?» C-suite and business leaders love this metric because it quickly shows whether engineers are working on what matters to the business.
Remember that it doesn't matter if you deliver more features if they're not the right features. You can use this metric too, as it helps you make a logical argument about how much work your team can handle. If you're asked to take on a new project, you can reasonably discuss with numbers whether we'll take on the new project or finish the old ones while we're reducing focus. This can also serve as a good basis for asking for more team members.
Metric 2: Project Planning Accuracy
Metrics «Accuracy of project planning» answers the question: «Will we be able to deliver on our promises for this project?» A recent study found that the average sprint planning accuracy among 1,900 teams using Scrum in 2021 was 46%.
You will never be able to deliver the promised features in a quarter if the teams involved in the project deliver less than half of what they planned every 2-3 weeks.
Metric 3: Speed of delivery of new features (Lead time)
Metrics «Speed of delivery of new features» (Lead Time) measures the time it takes to turn an idea or request into finished functionality available to users. This metric is important for assessing how quickly a development team can respond to changing business and customer requirements.
The Lead time metric is a key indicator of the development team's performance and its ability to quickly and efficiently meet the needs of the business and customers.
Conclusion
DORA metrics are not the end goal for the team in themselves, but rather should be interpreted as proxy metrics for achieving business goals.
However, it is also worth looking at metrics at the business/development boundary, which are true performance indicators:
Speed of implementation of new features (lead time);
General Velocity;
% fulfillment of promises or forecast accuracy;
% of resource allocation to projects.
You can learn more about metrics and practical tools in the framework practical online courses from OTUS. Go to catalog and choose the appropriate direction.
A great tool and platform for understanding team performance — Aimgerwhich we created to help teams and companies see blind spots and improve those indicators that ultimately affect business results, better justify decisions, negotiate with businesses, and optimize processes. If you want to learn more about Aimger, go to my telegram channel
It is also important not to forget that there are people behind the numbers, and that continuous improvement is only possible with the active participation of the entire team. It is important to create conditions in which developers can work effectively, contribute, and feel their importance in achieving the overall goals of the company.
Use metrics as a tool for understanding and optimization, not as the only criterion for success. And remember, the most important thing is to create value for the business and customers.
