Where's My Efficiency, Boss? How to use metrics in team management

Hello! I’m Sasha, and about 4 years ago I became the team leader of one of the primary real estate teams in Cyan, and before that, for two years, I wrote the backend in .NET. During this time, our direction has grown from a dozen people in one development team to 5 teams with more than 50 people in the direction.

My team is engaged in the monetization of developers – we provide tools for promoting developers’ content on the site, we give the opportunity to flexibly manage the advertising budget, and collect statistics. The team is flexible: we have pythonists, sharpists on the backend, front-end developers, and QA. Sometimes mobile developers drop by.

When I became a team lead, I needed to find guidelines. How do I know if I'm leading the team in the right direction? What was the effect of changes in testing processes in the team? Did I hurt quality by removing the mandatory weekly team meeting by an hour? Have we found the right bottleneck in the team's processes? It is quite difficult to answer these and other questions speculatively, without confirming them “with numbers.”

At Cyan, we actively use data to make management decisions. Not only product ones, but also when managing development teams.

In the article I will tell you how this works in our company, and I will show, using my metrics as an example, how you can use it in your own home.

What does it look like

At Qian, development team leaders use development metrics to analyze the situation and make management decisions. For example, we see one of the product areas on the metrics something like this:

How it works

The result of each developer's work falls into a number of different metrics. We display these metrics on dashboards and send pulses to channels in slack. For example, this is how we receive a daily summary of open bugs in the team channel.

We use the Metabase tool as a frontend for metrics. It allows you to flexibly configure different types of graphs, collect them into dashboards and make custom SQL queries directly from the UI. Under the hood, all this is stored in Postgres, so you can connect from a convenient tool and “twist” the data as you please.

We take data literally from everywhere: from Jira, from Git, from our CI/CD system, even from 1C we pull personnel information – vacations, holidays, etc.

Importing data is possible because our processes for working on tasks are standardized. All tasks with code go through Jira according to a strictly defined process. Bypassing this process, it will not be possible to deliver the code to production. Each type of issue has its own metadata, filled in manually (for example, an estimate in hours or T-shirts) or automatically (for example, the date of status change or the name of the repository in Git).

We upload all changes in the systems we are interested in into a self-written processor, which converts the incoming data into a digestible form. Next, team leads and projects look at existing metrics visualizations or do something for themselves. A new graph can be displayed by writing a query in raw SQL, or it can be configured through the interface.

What does this give us?

Here are a few examples where development metrics help me:

• Confirm or refute the hypothesis on team work. Tasks in one component take longer to complete than in others. Is this really true? How much longer on average? How many tasks did we complete there during the quarter? The team says that the difficulties are in running tests, but is this really true? I'll go and collect a summary of the team components, the number of tasks, their average rating, the number of bugs found and the time spent in different task statuses.

• Evaluate effectiveness implemented changes. In Qian, the rule “Wednesday is a day without meetings” was introduced. How did this affect my team?

• Look “on radar“how well and steadily we are going. After the release of a major project on March 1, the number of deployments in the team dropped by 2 times. Is it we who have “exhaled” and need to give time to recover, or is there some problem with new projects? We need to go figure it out.

If you look at real cases, my favorite example is the optimization of our Code Review throughout the company several years ago.

Then we noticed that after sending tasks to Review, it takes about 20–30 hours before the changes are sent to production. This is a lot, considering that new code in our CI/CD microservice can be ready to launch on production in a canary release in a few minutes.

It turned out that manually searching for reviewers, reminding and reviewing corrections is a simple but long process that depends on the proactivity of specific people. In addition, the load on code reviews was uneven – most of the reviews were done by several people as the most active and responsible. It took them a lot of time.

All this time, the author of the task cannot fully switch to the next tasks; often they are simply blocked by the release of the previous one.

After introducing automation into the review process, the time from review to release was reduced several times. Now more than 80% of tasks with code in Cyan get to production faster than 8 working hours.

This is all well and good, but the example is old.

What about current examples?

Example 1. Throughout the quarter, the team did 1 major project with a release by March 1 and a number of supporting tasks and projects as needed

Let's look at the graph of the number of product deployments by week in one of the teams since the beginning of the year.

The graph shows that at the beginning of the year the team “accelerated”, in the middle of the quarter (S6) it reached its maximum pace, and then it began to decline. Without context it looks the same.

The manager’s task is to understand whether everything is going well or whether some measures need to be taken? Preferably in advance. This chart is a signal to think about everything, and as a leader I see and understand everything in my team. If I can’t immediately explain such a state, this is a reason to delve into the details and understand what is happening.

As a result, everything is fine with the team here – at the beginning of the year they designed, negotiated and decomposed the main project. Then we parallelized the development and actively developed it. From Sprint 7, the project moved into the integration and testing stage, and in S10 – release support and the start of the next projects. S11 has just begun.

Example 2. The team's testing processes changed

Another team used a “standard” approach to testing – a combination of covering the task with tests by the developer and manual testing through QA. There were bugs in manual testing, and tasks were sent for revision, distracting the developer from the next tasks and slowing down progress on the project. This is most clearly visible in the graph of quick tasks. It can be seen that the target of 70%+ “fast” tasks was not achieved in every sprint.

Starting next year, the team introduced the practice of development testing – when the developer tests his task by hand, without involving QA. Essentially, different parts of the project are now tested by developers, and QA is responsible for the end-to-end integration of the project.

This change speeded up the delivery of tasks to sales, because they stopped actively re-opening and finishing tasks after the Code Review stage.

The development of the tasks themselves was not accelerated, sometimes even slowed down, because the developers also began testing. But now these tasks need to be kept in focus less after the development stage is completed and you can be more actively involved in the next ones.

Another consequence is that more bugs began to appear on the product, even Major bugs appeared. This is understandable – the developers are not so experienced in testing, but this effect has been smoothed out over time. Neighboring teams have similar experience.

Example 3. Summary of work done

The screenshot shows a summary of the two direction commands. This summary is useful to summarize the work done over the past 2-3 months.

Not a silver bullet

It is important to understand that development metrics will not solve all problems for you, and can also add new ones – it is still a tool, and it must be supported. From experience with our metrics, I drew the following conclusions.

Very important don’t make metrics an end in themselvesotherwise, in our desire to get to some number on the graph, we will forget the original goals and will not get the desired result in the product.

It is necessary to carefully test collect data and check metrics for correspondence to reality. Several times it happened that the graph looked good, and after a while it turned out that it showed something completely different from what we thought.

Healthy recheck your hypotheses from metrics in practice and vice versa. It seems that you are starting to do few tasks? Or maybe some of the tasks were simply created incorrectly and ended up in another team?

When analyzing indicators it is worth take a wider view, taking into account neighboring metrics. Has our speed of delivering tasks to sales slowed down? It is worth looking at what were the reasons for the reopens. Or maybe the team was unfocused due to a surge in bugs?

Total

Metrics help you better navigate what is happening. With their help, I find confirmation or refutation of some of my hypotheses on command. Metrics allow you to find and treat problems in a timely manner. With the help of metrics, I can set more specific goals and also track progress on the changes being implemented. Without metrics, I would do everything on my own “feeling,” which could have a completely different effect on the team.

If you want to know the implementation details, ask questions in the comments. And tell us what metrics you use in your teams.