Product, cases on analytics from Yandex. (Distinguishing good metrics from bad ones)

“Analytics is like a matan. You have to solve a lot of tasks a day to gain experience and visibility” – Dmitry Timko, head of Yandex Browser.

I continue to work on product analytics, I came across a very high-quality Lecture by Dmitry Titko at Yandex School of Managers. The author gives detailed cases of choosing metrics for different products around Yandex Browser. Explains how feature metrics are related to overall product metrics, how to distribute users across samples, and why a good metric can turn out to be bad.

The material turned out to be very interesting and practically useful. There are few simple and correct answers here, and many cases to think about. Then I share my reworking of the lecture.

Theory 1. A/B experiments

  • Randomly recruit two equal user groups

  • We distribute an experimental improvement to one group

  • Compare the selected metric at the end

Hitting A/B selections should be random. There should not be such a sign that the user got into group A and cannot get into group B, as in the example below.

Example 1. Browser login

There is an assumption that a logged-in user has more engagement. Let’s increase the login. How to test a hypothesis?

  • We take 100,000 users, see how many of them are logged in, let’s say 25,000.

  • We take another 25,000 from the same sample who are not logged in and compare

  • The second group will actually perform 10% better, is everything okay?

Where is the mistake:

  • People who logged in could be initially more motivated, so they performed better.

  • That is, users are in group B not by chancebut on the basis of the fact that they are logged into the browser.

The result cannot be considered reliable.

Theory 2. Rules for selecting metrics

  • You must select one metric that you will look at at the end.
    Yes, there may be boundary conditions, but it is important to choose one point. It can be a combination of metrics.
    If it is important to increase one without losing the other, track the sum (difference) of the metrics.

  • We check whether it is possible that we have improved the metric, while in fact worsening the product.

  • We define a list of other features (elements of the ecosystem) of the product that may suffer from improving our feature.

Example 2. Browser promo page

Improving the promo page of the browser.
The page has a USP and a download button. At first glance, the task seems to be:

Why is this a bad metric? By the fact that you can download, but not install. The product does not need downloads, but installations. Installs are not a good metric either. Can be installed but not used.

For the product as a whole – the Browser – it is important not to optimize the promo page, but to raise the product metrics, what metrics these can be:

From the proposed ones, Dima chooses the Usage metric (average time) by the fact that:

  • Usage is not suitable because by attracting users with a large Usage, you can lose users with a smaller one, the Return will decrease, and the Usage time. (To combat this, you can segment users and roll out the improvement only to those who need it. )

  • DAU is bad because it falls during the low season, which does not mean that the product does not work well. It’s just that in the summer people spend less time on the Internet.

  • Time is a good revenue proxy in the case of a browser.

In the case of the Browser, there is an even better metric:

  • 🥇 Total visits to sites from Yandex Browser.
    This is the share of all pages on the Internet opened by Yandex browser, compared to other browsers – there is a service RadarHe shows.
    (I marked the metric with an icon for convenience, because we will return to it later in the article)

Back to promo page

  • We can say that a promo page is better when more people who visited it and installed the product did more useful actions in the browser.

(NB) Linking a feature to high-level product metrics is not always possible. Metrics may not “paint over” for a long time (do not show statistically significant changes).
In this case, you will have to come up with a narrower metric for the feature itself. If it’s super difficult to come up with a narrower metric, and general metrics are not painted over, you may have to abandon the feature altogether 😢

A note about recurrence.

There are different types of retention – rolling retention, churn, x-day retention. As part of the lecture, we will use the returns of the 2nd week. How many of those people who delivered the product 2 weeks ago used it.

Example 3: Weather Push

Yandex has cool weather. We decided to try to show a push with the current weather to the user on the splash screen. How to measure the effectiveness of a feature and why is retention a bad metric in this case?

  • The idea of ​​pushing is to make a difference and grow retention through the user’s love, and not because of accidental/impulsive clicking on push.

  • A push with weather info is a product that you don’t have to click on. All data is visible at once.

Introduced updated retention.
They began to count only those users who saw the push, but:

  • Didn’t click on it at all.

  • They pressed, but were not limited to viewing the weather, but did their other things. (the user was going to work in the browser anyway, the push just speeded up the start of the session)

If such retention grows, then the push brings benefits and increases loyalty. As you can see, you have to dance with a tambourine with retention, so the general metrics are better:

Example 4. Working with backgrounds

  • Maybe measure how many people came to the gallery? A good metric to start with, but I want something closer to the target action (webviews through a browser)

  • Maybe again the general metrics of the product (usage, retention?) They were not painted over for a long time. You need to choose the metric of the feature itself.

  • Maybe the number of background changes per day?
    It may be that users change the background that they did not like, the metric grows. Is this good or bad?

  • You can try to measure what kind of background the user set, and whether he returned to the previous one.
    At this point, we must admit that we have entered into the wilds. Our study design is shaky and time-consuming. We’re going to do unbearable things.

  • We decided to return to general retention. Yes, it will take a long time to stain, or it may not stain at all. But since the feature is so complex, we will try to evaluate its impact on retention.

  • If there is no influence, we will not deal with features at all.
    Because a feature at some point can become a deterioration that we will not be able to notice.
    (in a few years will gain a cumulative negative contribution to the product)

Example 5. Zen in the browser

Added news and weather to the main page in the browser, how to measure the effectiveness?

Total referrals and returns have increased, but what’s the problem here?
Yandex has other products that have been affected.

  • People began to consume news and weather on the main page of the browser, and, perhaps, stopped coming to the Yandex main page for this.

  • But there are other features on the Yandex home page, such as emergency alerts and other social media. a responsibility.
    Overall product metrics could go up, but the homepage team is unhappy. By improving our feature, we have greatly reduced the use of another service in the ecosystem.

As a result, we added several widgets to the main page of the browser leading to the main page of Yandex and everyone is happy.

Example 6. Offline copies of pages

If you read some page (on mobile OSes), then turned it off and open it after a while, the page is loaded again. And if it’s a longread, you’ve lost the scroll you were on. And if it happened on the escalator, and if it happened on the plane? We decided to save any open page on the device. How to measure improvement?

🥇 Total visits to websites?

  • The problem is that the pattern of behavior is changing. The user, realizing this feature, can open bookmarks before entering the subway to read later. For the share of users, total clicks will increase.

  • On the other hand, in the past, some pages had to be opened twice. And with this feature, re-downloads are not required, the total transitions will fall.

  • Another problem is that offline views do not generate ad impressions, which penalizes content creators. When improving a product, you need to think about the benefits of the entire ecosystem. While trying to improve the experience with offline browsing, you need to improve the online experience as well.

As a result, we decided that it is important for the feature to use it consciously, which generates the opening of a large number of tabs, rolled out a tutorial, made the work of the feature more visible.

That’s all, I personally liked the lecture very much, I hope it was useful to someone.
If it’s more convenient for you to read me in a cart, here is the link to the channel.

  • e Usage – Average usage time

  • DAU – Daily Active Users

  • Retention – Returnability

  • 90 day return

Similar Posts

Leave a Reply