perhaps the best way to understand what is wrong with your product

Hello! I’m Lena Mayorkina, I work as a CPO in AGIMA. We’re doing cool stuff here: developing products for big companies and coming up with plans to take over the world. To succeed in both directions, you have to work hard with metrics. We analyze which metrics are relevant to the product, where its strengths are, and what needs to be improved. And our main tool is the Metrics Pyramid framework. In this article I will explain why we love him and why you will love him too. If you are a product or analyst, then the text is definitely for you.

When we want to make a product successful, the first thing we need to understand is the criteria for success. In our case, these criteria will be the values ​​of the metrics. Proper development means proper metrics: relevant, comparable, understandable, measurable, and changeable.

We monitor metrics, manage their value, and as a result, this affects business performance. Sometimes the correlation is direct, sometimes not so much, but it is always there. Therefore, it is important to remember exactly those metrics that provide value to the user. The light did not converge on Retention and LTV.

To find out which metrics are relevant to a product, the Metrics Pyramid framework is the most suitable. This is a cool tool, often it lives only in the heads of product owners and is not available to the product team. At the same time, there is no universal pyramid of metrics, everything always depends on the business and the product.

Pyramid of Metrics and Hierarchy of Metrics

The pyramid of metrics and the hierarchy of metrics are often confused. They are really similar: both are hierarchical models. But here we will focus on their differences.

In the Hierarchy of Metrics framework, we decompose a key success indicator into metrics that directly affect it. They, in turn, are even more precise. And so on. Hierarchy helps you organize metrics, identify dependencies, and understand which metrics you need to actively work on to achieve the most significant result.

As a result, we:

• we understand how changes in processes affect the improvement of product performance (in dynamics, including the long-term effect);

• we know which indicators can worsen when the selected metric improves;

• we see order in the backlog and simplification of task prioritization.

Clearly, a hierarchy of metrics is useful. But in this framework, we cannot always understand which metrics are more significant, which are intermediate, what should be focused on in the next sprint and at the same time globally avoid hyperfocus.

The pyramid of metrics will help with this. We distinguish four levels, which are located from micro-processes (platform metrics) to macro-structure (business metrics). I will talk about this below.

Polaris metric

Let me remind you what the North Star Metric is (the metric of the polar star, the metric of omnipotence). It is believed that NSM is an indicator, by monitoring and influencing which, the company will definitely achieve its goals. Sometimes this metric is called a utopia, but often it works.

But this approach has drawbacks. Take, for example, a business checking account. Consider two metrics: the number of new customers (focus on marketing) and the number of monthly active customers (focus on the product). What’s more interesting? Depends on goals. And if we take Revenue as the omnipotence metric, it will be universal for all products, but will not help much in terms of decomposition.

A pyramid of metrics is needed to prevent hyperfocus and keep abreast of the product and business, and the hierarchy of metrics often works the other way around. It’s like if the patient is treated by a doctor who is sure that if the temperature reaches 36.6, the rest of the symptoms of the disease will pass by themselves.

How do we build a pyramid of metrics

We don’t get hung up on NSM and its components. Working with such a pyramid is possible only with auxiliary and control metrics, which somehow makes such a decomposition a waste of time. We consider NSM as an indicator of value for customers, therefore, we use performance and added value metrics as NSM. Increasing the added value of the product is the main task of the product team, and product metrics cannot always help with evaluating the effectiveness of solving the client/user problem.

Measuring the values ​​of such indicators is not a trivial matter. So here we will just limit ourselves to top-level definitions:

• problem solving efficiency metrics show how much effort and resources are required to solve the user’s problem or achieve certain progress within its solution;

• Added value metrics show how much more effectively one product solves a problem than another.

Accordingly, we build the pyramid of metrics a little differently:

• during the audit phase, we determine the data that is tracked and that needs to be tracked;

• then we classify the indicators in order to get rid of hyperfocus;

• then add the base hierarchy;

• building links between metrics.

The result is a pyramid of metrics that helps prevent hyperfocus. And this pyramid consists of four main “bricks”: platform, interface, product and business metrics.

Platform metrics

At the base of the pyramid are metrics related to the availability and technical reliability of our product. If the product cannot be used “for technical reasons”, then there will be nothing to measure.

Interface

Next come the interface metrics that show the user’s interaction with the product. This includes the effectiveness of advertising campaigns, and conversions of forms that the user fills out, and conversions of buttons like “leave a request”.

Grocery

They characterize the behavior of users and the economics of the product, answer questions about the product itself. They allow you to understand how the product turns new users into other entities.

Here are some examples:

• Retention shows how new users turn into active ones;

• LTV shows how new users turn into profit for the entire time of using the product;

• First purchase conversion shows how a product converts new users into paying customers.

Business (they are growth metrics)

If product metrics describe the product itself, then growth metrics describe the business that is built around this product, show the final result of the transformation of new users using the product into other entities. That is, grocery – HOW trousers turn into, and business – into WHAT they turn into. (In smart shorts, yes.)

Examples of growth metrics:

• DAU or Daily Active Audience (New Users * Retention).

• Profit (New Users * LTV).

• Number of users sending messages (New Users * Retention into Sending a Message).

• The number of new subscribers (New Users * Conversion into Subscriber).

• Revenue, Profit, margin, sales volume, etc.

When we evaluate product changes, we cannot focus on growth metrics, since these indicators depend not only on the characteristics of the product, but also on the influx of new users.

What’s next?

When we have built the pyramid of metrics, checked the metrics match with our business model, and studied the metric values, the fun begins. This is working with metrics and monitoring their changes, predictive analytics and the impact on achieving the desired indicators – that is, the constant development of the product and, as a result, the business.

Pyramid of Metrics and OKRs

The combination of OKRs (goals and key results) and a metrics pyramid is a powerful tool for a product team. Now I will explain why this is useful.

– Target alignment. OKR helps the product team identify the main goals and expectations associated with a product or feature. The Metrics Pyramid translates these goals into measurable performance metrics that can be tracked and analyzed. The combination of OKRs and the metrics pyramid helps the team understand how their work affects the overall goals of the organization and ensures alignment at all levels.

– Measuring progress. The Metrics Pyramid provides the team with a set of performance measurements that reflect various aspects of a product’s performance or functionality. These metrics allow the team to track their progress and evaluate how well they are achieving their goals. OKRs, in turn, provide clear and specific results that the team is striving to achieve. The combination of OKRs and a metrics pyramid helps a team measure and demonstrate their progress towards achieving goals.

– Focus on key metrics. The Metrics Pyramid helps the team identify the key performance metrics that are most important to achieving product or feature goals. Focusing on these key metrics will enable you to focus on the end results and make informed decisions to improve these metrics. OKRs help the team prioritize and direct work, while the metrics pyramid helps the team focus on those metrics that are most essential to success.

– Adaptation and improvement. The combination of OKRs and the metrics pyramid allows the team to evaluate their results and tailor their work according to feedback and performance measurements. A pyramid of metrics helps you identify weaknesses and find opportunities for growth.

– Continuous learning and improvement. OKRs and the metrics pyramid encourage continuous learning and improvement within the product team. Regular review of performance metrics allows the team to identify areas for improvement and take appropriate action to optimize the product. The team can use the data and results to make better informed decisions, experiment, and make changes to the product to achieve better results.

Let’s take an example. We have a business goal to increase website sales by 30% in the next quarter. To achieve it, we set OKRs – increase the conversion on the site from 2% to 4%, as well as increase the share of search traffic by 15%. To measure progress towards these outcomes, we can use a metrics pyramid. At the top level of the pyramid, there will be general business metrics such as total sales and total revenue. Below may be metrics related to marketing channels, such as the number of clicks on an advertisement and the number of new visitors to the site. Next, there will be more detailed metrics related to specific user actions on the site, such as the number of additions to the cart, the number of completed orders, the number of abandoned carts, etc. To measure progress towards a key result, increase the conversion rate on the site from 2% to 4%, in addition to tracking the conversion itself, we analyze the values ​​​​and influence a number of related metrics that reflect user behavior. Thus, by reducing the value of the bounce rate, working with the number of abandoned carts, etc., we are moving towards our key goal, while fixing the result.

Predictive analytics

Here we will dwell a little more on the topic of predictive (predictive) analytics. If we can predict the expected profit, churn and other indicators, then it will be possible to develop the product much more efficiently. In order to do fortune-telling on metrics, you need to understand what it is all about.

Predictive analytics is the process of using data and statistical models to predict future results and user behavior in the context of a product. Key aspects of predictive metrics analytics:

1. Data collection.
   Predictive analytics requires access to historical data about the product, that is, for all the components of the blocks of our pyramid in retrospect.

2. Identification of target metrics.
   Determination of key indicators, which for us are now the main ones for evaluating the effectiveness and success of the product.

3. Data modeling.
   Once data is collected and aggregated, it is run through ML algorithms for analysis and prediction. Model examples include linear regression, time series, decision trees, and other methods.

4. Forecasting and optimization.
   Forecasts can be presented in a variety of forms: numerical values, graphs, predictive ranges, etc. It all depends on what you agree with your Data Analyst. Optimization is based on a cycle of experiments and continuous analysis of the results to test the effectiveness of the proposed changes.

5. Estimation of model accuracy.
   Comparison of the actual values ​​of the metrics with the predicted ones makes it possible to evaluate the accuracy of the predictive models. This includes calculating accuracy score metrics such as MAE (mean absolute error), MSE (root mean square error), coefficient of determination, and others. Evaluating the accuracy of models helps determine their reliability and applicability in real-life scenarios. As data on specific indicators accumulate, the models make more and more correct forecasts.

6. Making decisions.
   Based on the forecasts, it is possible to determine product development priorities, investments in marketing, user experience optimization and other strategic activities.

It looks like it’s not easy. Therefore, it is more convenient to use a change prediction calculator.

Change Prediction Calculator

The Change Prediction Calculator is a tool primarily used in Digital Marketing to estimate planned changes in the effectiveness of an advertising campaign. A simple example is “Budget Forecast” in Yandex.Direct. But it can be used not only in marketing, it is also possible to predict changes by metrics.

The principle of operation of the calculator for predicting changes in effects by metrics may vary depending on the specific case, but it is always based on statistical methods, models and ML algorithms. Working with such a calculator (regardless of the source of its receipt) will be based on these steps:

• data input;

• data analysis;

• forecasting changes;

• visualization of results;

• interactivity (optional).

The calculator works on the basis of user-provided data and can predict how key metrics will change with various changes in specific parameters.

However, do not forget that it cannot take into account all the factors of influence, therefore, the results obtained using the calculator can only be approximate. You should always consider real-world conditions and observe real-time metrics to more accurately assess predicted changes.

We live in a never ending HADI loop for constant scaling and use a variety of tools at every step. But that’s a completely different story. Colleagues sometimes share parts of it, for example, about dashboards and their help (or not) in making business decisions, here you can read about qualitative research, and Here about quantitative – working methodologies for the stages of formation and validation of hypotheses.

I talk about individual stages and methods in the telegram channel “Product Effect // Lena Mayorkina’s Channel”. Subscribe and ask me questions there. I will write more.