Obsession with controlled input metrics

This translation was created for the telegram channel Data Engineering… And, since, one of the tasks of the creators of the telegram channel and the analytical course DataLearn is to popularize competent work with data, we post translations in the public domain. This translation takes a closer look at the concept of Amazon’s controlled input metrics and its benefits for your business.

I admit: I’m a little obsessed Amazon’s concept of controlled input metrics (translation).

The idea itself seems extremely simple. “And?” I hear you shout, “Isn’t controlled input metrics just a tricky way to say”leading indicators”? And isn’t this just something that everyone should already do when exhibiting OKR? ” These are good points. But they go against the style of thinking that controlled inputs will force you to adhere to to a large extent. I would argue that supervised input metrics are the type of ideas where methods seem trivial at first, but after putting them into practice, the way you think about the data changes.

In fact, I’ll even go further and say that they have fundamentally changed my understanding of operational efficiency.

I would like to give you the opportunity to experience it in this post. So let’s get started.

try it

The key to understanding the nuances of this idea will be trying to derive controlled input metrics for your company… Respect me. Think about any business process in which you are involved. Now think about the metrics you can measure to measure this process.

If you are like me, then you would come up with such metrics:

Sales: The number of trades closed.

Marketing: Blog traffic or weekly number of new weekly subscriptions.

Software development: the number of new features delivered over the period.

Customer Support: Survey Results After Interaction with Customer Support /NPS…

Etc.

The chances are that the metrics you choose will be outputs, that is, the results you want for the business, but over which you really have only indirect control.

For example, blog traffic. Let’s say your goal is to double your blog traffic over the next 6 months. What would you do? Would you write more guest posts? Would you attract more SEO traffic? Would you like to tidy up the structure of your site? Would you try to make some viral posts?

The likely answer is that you will try all of these things, knowing, however, that you will have a delayed effect on your efforts. For example, SEO-related improvements can take on the order of 6 months from actions to results.

So that’s why measuring the output is not good enough. It is not enough to motivate your employees and certainly not enough if you work for Amazon.

How is talking Colin Briar, if you want to be a good leader, you need to have a very specific understanding of the factors that you can control that directly affect the output metric you care about. And you need to measure them. He argues this as follows:

“The best managers I’ve seen are very clear about which ‘buttons’ to press and which ‘levers’ to use to get the result they want. They see right through the process. “

These input metrics are usually associated with users. “Is the customer experience better this week than it was last week? This is harder to know than it sounds. So you monitor 10 or 20 different things with a little experimenting. ” Says Briar. “Evaluate them day in and day out – a great manager always measures, so he knows exactly what is going on. If you are not evaluating something, it will most likely go wrong. “

“The best managers I’ve seen are very clear about which ‘buttons’ to press and which ‘levers’ to use to get the result they want. They see right through the process. “

These input metrics are usually associated with users. “Is the customer experience better this week than it was last week? This is harder to know than it sounds. So you monitor 10 or 20 different things with a little experimenting. ” Says Briar. “Evaluate them day in and day out – a great manager always measures, so he knows exactly what is going on. If you don’t evaluate something, it will most likely go wrong. “

What you will now want to measure is monitored inputs – things that you know can be improved today that will lead to changes in the desired output metric in a few months. These inputs tend to drive employee behavior if you set them as organizational goals. For example:

For sales, you measure the number of responses / interactions with a unique user that matches your ideal customer criteria. (Target output metric here: number of trades closed).
For marketing:% of blog posts with a personalized and attractive subscription form embedded in the text. (Target output metric here: number of new mailing list subscribers).
For software development: the average time from commit to deployment. (Target output here: number of new features delivered over the period).
For the customer support department:% of tickets that were closed within 3 days. (Target output is here: NPS).

A key property of many of these metrics is that you stop and say, “wait a minute, I don’t think these metrics will lead us to the desired outcome.” And that’s the whole point. The monitored input metric is, by definition, several layers lower than your expected output metric, and you will need to find out if there is a strong relationship between the two. And more importantly, a controlled input metric should provoke changes in the work within your team, and you should feel fear that this will lead to some kind of inconsistent changes (think about adverse situations from Goodhart’s law).

Amazon argues that this is completely predictable, and this is why it is necessary to go through a period of trial and error to see if the input metric leads us to the desired results or not. A huge part of my synopsis Working Backwards tells about Amazon’s processes for choosing controlled input metrics. They call it “walking through the metrics lifecycle,” the process of selecting metrics is called DMAIC (“Define, Measure, Analyze, Improve and Control”), or “Define, Measure, Analyze, Improve and Control.”

So: I hope you are still with me. I hope you have tried choosing a plausible input metric for your business process, as well as an appropriate output metric that it should influence. I want to provoke you a little more: try to scroll through a few processes in your head in your business. For each of the business processes, ask:

What is the expected behavior we want to see here? (For example, for a data processing team, this could be: “we want to increase the percentage of business decisions made using data”).
What is our output metric for this and how do we measure it? (How are you going to record whether decisions are made using data or not?)
What is our input metric? Potential Candidate:% of internal reports used – whether through a dashboard or reading an email – at least once a week.
Now get ready to pass DMAIC trial and error processwhich Amazon uses to check if the selected input metric actually affects the output metric.

Switch to time, and then repeat these questions for the rest of the processes you want to work through in your head.

Why is it so powerful?

Let’s go back to my original question: why is this important? And why does this idea seem so great to me?

Нас не учили так думать

The simple answer is that we weren’t taught to think that way. When people say “base decisions on data” / “make decisions based on data,” we immediately think, “oh, we need to measure our business results.” And so we measure things like closed deals, cohort retention, revenue growth, or blog visits. These are all output metrics, and while important, they don’t really work.

Amazon says it’s not enough to know the output metrics. In fact, they go even further and say that you shouldn’t pay too much attention to the output metrics; you should keep an eye on the controlled input metrics that you know affect those very output metrics. It’s completely different when, say, your customer support team knows that their premiums are based on a combination of NPS and% closed tickets within 3 days. If you could clearly demonstrate the relationship between the first and the second, then each of the team would be motivated to come back with an optimized process just to increase that percentage!

Of course, there are potential problems with this approach. But Amazon has developed a number of techniques (translation) to prevent inconsistent changes:

Each metric has its owner, and each owner of the metric must understand what is the norm and what is anomaly.
Metrics are audited by an independent (finance) department, which prevents product owners from changing metrics or choosing metrics that flatter executives.
The metrics are reviewed every week at the weekly business review meeting. (This happens fractally – from the top management level down to each team).
The finance department informs both senior management and product owners how they track the achievement of each of the annual targets, both in terms of inputs and outputs.
Executive directors and product owners are expected to critically assess the performance of each metric and are allowed to modify and remove metrics if they are no longer useful.
Metric owners should have a process in place to audit each metric to ensure they are measuring exactly what needs to be measured. The audit process is started at regular intervals.

What I’m trying to emphasize is that Amazon uses metrics as a finely tuned operational weapon – they understand, in terms of developing incentives, that a well-designed metrics framework influences the behavior of an organization.

In the world of business intelligence, we often spend our time discussing cool cutting edge tools, new tweaks in the pipeline, unit tests, or “notebooks, not dashboards!” But it probably doesn’t matter if your organization is not tuned in to make data-driven decisions.

Amazon held its first metric meetings using stacks printed sheets… And they did it in the early 2000s, when columnar DBMSs were just emerging, and OLAP technology was still the norm. But the result surpassed the wildest dreams.

So: what can we say about ourselves?