when recorded improvements lead to deterioration

Hi, my name is Alexandra Khoroshkova, I am a project manager for mailing lists at SuperJob. Millions of users receive our emails every day. This audience provides an excellent opportunity for quick testing of hypotheses, since it allows you to conduct the required number of experiments in a short period and understand whether there are statistically significant differences between different options.

In this post I would like to share my experience in optimizing our most massive mailing list – “Hot Jobs”, to which 5.5 million users are subscribed, that is, almost every our active applicant. More precisely, a case when a series of positive changes can unexpectedly lead to a negative result, and what needs to be done to overcome this.

Collecting traffic from the negative

At the very beginning, before all the changes, “Hot Jobs” looked modest and discreet (left). But pretty soon, after a series of tests, we brought the mailing list in a slightly different form (on the right):

We did not stop there and began to improve our “Hot Jobs” further. The idea came up to collect traffic from the negative – from offers that the user is not interested in. For example, if the recipient of the letter has already seen the vacancies offered in the mailing list in the search results on the site, then he probably will not go to look them again. But he may want to block them or hide them from search results. To do this, he will have to go to the site, and there there is a chance to get involved in some of our other scenarios: see similar vacancies, read the notification feed, and so on.

We have developed and started testing three variants of the button:

According to the test results, the option with the “Block” link won:

Our letter now looks like this:

Drawing attention to video vacancies

One of the current trends is to read less, watch more videos. Therefore, the number of video vacancies on our site is growing. Of course, we wanted to emphasize them in the newsletter as well.

We came up with several options for testing and, as a result, chose the one where the “Video vacancies” mark is located right above the title of the job offer.

So, we worked with the appearance of the letter, collected traffic from the negative, and highlighted video vacancies:

What else could be improved?

We decided to test the hypothesis that users are more willing to respond to fresh vacancies – published in the last 24 hours. Moreover, such a feature will be useful both for the applicant, to whom we highlight the most “fresh”, and for the employer, who receives responses on the first day after posting his vacancy.

These are the ways we decided to highlight fresh vacancies. It highlights which accents we tested:

As a result, the option with a call and a “Apply” link, placed under the vacancy on a green plate, won.

Better – easier?

Let’s take a look at what our email looks like after a year of consistent optimization:

From a simple and concise message, the mailing list has evolved into a letter overloaded with accents and some kind of color spots. At the same time, it became more difficult for the applicant: an excessive number of parameters makes the choice difficult, not easier. Unsurprisingly, email conversion inevitably began to decline, so we started testing new lightweight versions of the vacancy snippet (preview) in the email. Something was removed, something was moved – and we got three new options for testing:

In terms of the number of clicks to the job description, the option with the “View” link won, that is, calling for a click. And according to the responses – the option with the link “Reply”, although it got fewer clicks. But the option is completely without call to action (call to action) our users did not enter.

Since our key metric is responses, we have left the option with the “Apply” link. This is how it began to look in the letter:

Below is a fragment of the graph for the observation period from May to August.

We noticed that after the May holidays, the graph with conversion rates slid down. Usually, there is a seasonal decline by summer, but this time it was sharper (1).

At point (2), a test of lightweight writing was launched and the graph crawled upward, and at point (3), the winning option was rolled out to the entire audience. As a result, according to the responses, the test variant gave 6% more than the indicators in the control variant, and 4.8% more clicks. Of course, it’s not just our tests that affect chart fluctuations, but the correlation is definitely visible.

Results and conclusions

As you can see, the letter has become not only visually “cleaner”, but also shorter. Now the user does not need to scroll so much to reach the end of the list.

We are now testing these snippets in other emails to clear them of “visual noise” too. For each mailing, you need to check the hypotheses again: just take the experience of “Hot Jobs” and apply them to other areas without A-B testing. Why? First, each email is sent in a specific situation with a specific set of rules. Secondly, if the key metrics in the case considered today for us are responses, then in other cases it can be creating a resume or going to a website.

Nevertheless, the experience of optimizing the most mass mailing of SuperJob has taught us a lot. Here are the main lessons we learned from this case:

Good results taken individually are not always good overall. Try to see the full picture of the world and watch how each change correlates with each other.
Don’t stop at all. You should always experiment, try, learn from the successful experience of your colleagues.
Don’t be afraid to undo your changes, no matter how successful they seem. Sometimes rolling back can be a big step forward.
It is important to objectively assess your audience, to understand that it also changes and sometimes “burns out”. The effect of novelty goes away, and the user is not interested in your changes.

That’s all, I was glad to share my experience with you. Ask questions in the comments. It is possible not only on the topic of the article, but also in general about mailings and their monitoring in SJ.