How many students actually find jobs after Skillbox?

In 2021, I began a large-scale employment study of 400 QA students in more than a dozen different courses. The study turned out to be so long that it turned into a dozen habr articles with 500,000 views, and some of its participants managed to go from “what course do you recommend?” to “congratulations, I have been offered the position of senior.”

As it turned out later, I was not alone in researching employment statistics. In 2021 and 2022, the Higher School of Economics conducted research on achieving student goals for several large learning platforms.

IN publicly posted report There is data on real employment after training in Skillbox to become a tester. They do not lie on the surface, but it is not difficult to isolate them from the report.

Surprisingly, this proprietary data looks different from the numbers that Skillbox publishes on its landing pages.

Study “Monitoring the achievement of learning goals for Skillbox graduates”

This study of the effectiveness of Skillbox educational programs was conducted using a standard methodology for the Higher School of Economics (HSE). According to its description, the course transmits primary data to HSE, and it processes it.
Based on the description of a similar study for Yandex.Practicum, the course independently contacts all students:

About 500 graduates of a dozen Skillbox courses in various professions were selected. These graduates were asked to complete an online questionnaire. It contains quite a lot of questions that determine what goals the graduate set for himself and to what extent, in his opinion, he managed to achieve them. It turns out not only the fact of getting a job, but also how the graduate looked for it, how long it took and how it affected wages as a result.

Other things that are more interesting for the course itself are also assessed. For example, how often do graduates recommend training, how adequate, in their opinion, is the cost of training, and so on.
All of the above is collected by gender, age, place of residence, and previous experience in IT. As we can see, during the study, both objective and subjective data are collected and analyzed.

What the numbers say

60% of students reach the end of their studies. Don't look for this number in the report; I'll tell you where it came from a little later. In general, the report does not detail all the parameters for each profession, so somewhere you will have to use data generalized for all professions.

66% of graduates set themselves the goal of finding a job in a new profession. However, during the learning process, some changed their minds about changing it, and thus 62% remained.

Among these same 66% of those who wanted to find a new job, it was in the context of the specialty “Web Application Testing” that 66% were able to achieve their goal. Interestingly, from the calculation of the percentage of those employed, those who, in principle, wanted to find a new job, but did not begin this search within six months from the date of completion of the course, were removed. We will also talk about this later.

So what do we get? Of those who entered the course, 60% of students complete the course, of which 66% said that they wanted to get a new profession, and of these, 66% succeeded. Using simple calculations we get 60%*66%*66%=26%.

In other words, out of 100 students who entered the course, 60 completed the course. Of these 60, 66%, or 39 people, wanted to get a job as a tester. And out of these 39, 66% or 26 people were able to get a QA job.

Watching your hands

If we look again at the landing screen above, we see “93% said Skillbox helped them achieve the goal they set for themselves.”

If we look at the research, we see “66% of those who wanted to change their profession ended up finding a job.”

That is, even if we look only at graduates, and not at all applicants, then in fact the rate of goal achievement is 1/3 lower than stated.

But it makes sense to look at overall employment data, taking into account all applicants, not just those who graduate. Then we will see 26%. Not a very impressive result, right?

However, this figure is probably even lower. After all, there are some questions about the research methodology.

Questions about the research methodology

“66% wanted to find a job in a new profession”

Is this percentage too low? Pay more than 100,000 rubles for training just to learn something for your own satisfaction, but not to earn money?

Perhaps the rich have their own quirks.
Or maybe the fact is that we change our minds very often.

Sometimes we do this unintentionally. In my research, where I communicated with respondents – students of courses – from the very beginning, there were those who, during in-depth interviews, sometimes extending for a year, unnoticed by themselves changed their opinion from “yes, the profession of a tester is what I I want” to “I never intended to work in testing, I studied for myself.”

Sometimes we do this on purpose. “I paid for the course, but as a result I couldn’t find a job. And my friends got jobs in my study group. In order not to look like an idiot, I’ll point out that I didn’t initially intend to get a job.”

Let me remind you that in the HSE study, students were surveyed not just after completing their studies, but when more than six months had passed since graduation (and a year or more since the start of their studies). In such a situation, many students will obviously lower their original expectations in hindsight. What can give more positive statistics for the course is “they didn’t want to get a job in the first place.”

“We gentlemen take your word for it.”

Another important point is that the study does not verify employment. That is, if a graduate indicated in the application form that he was employed, then he was employed.

Some of the example above may not be comfortable admitting that they have not achieved as much success as their classmates.
Some student may be a Cossack sent by the course itself to boost statistics.
And someone from the example below might do this out of gratitude for the course.

To compare approaches, in my audits I explicitly require proof of employment from each employed graduate. This could be confirmation through working colleagues, a manager, or sending a letter from a business email.

Twenty dollars is twenty dollars

The invitation to participate, which was sent to graduates by email, described the purpose and topics of the survey, information about the approximate time to fill out the questionnaire (5-10 minutes), as well as information about the reward for filling out the questionnaire to the end – a certificate for one e-book, a link to which was placed on the last page of the questionnaire.

How can you ensure that gifts or rewards for taking a survey do not skew survey results by motivating people to give expected answers rather than their true opinions or experiences?
In my humble opinion, no way.

Carelessness with data

In a similar study by the Higher School of Economics on the employment of Yandex.Practicum graduates, it is stated that the online survey was conducted among graduates who completed their studies in the period from January 1 to April 31, 2022. It's just a pity that April 31st doesn't exist.
And it’s a pity that this slightly reduces the level of confidence in the data from other reports, including Skillbox.

However, this is far from the most significant.
After all, we have much more significant questions.

Why does everyone pretend that dropouts don't exist? Because the more students who drop out, the higher the percentage of graduates who are employed

The main complaint about the methodology to HSE is that it studies a lot of different data in detail and scrupulously. But by accident or on purpose it does not take into account the most interesting things.

Let's imagine that the same technique is used to assess the satisfaction of patients in the surgical department of a certain hospital. We survey patients after surgery about how satisfied they are with the treatment. How do they evaluate the surgeon’s work, how long did the recovery period last, and whether their quality of life changed for the worse after the operation.

Surprisingly, with this method of assessing satisfaction, the work of a bad surgeon can be rated higher by patients than the work of a good surgeon.
Why?

A good surgeon saves all patients. Some of them were seriously ill, faced with difficult illnesses, and their recovery is difficult. And even taking into account the fact that they were returned from the other world, one way or another they will have the thought “maybe the operation could have been done better?”

Patients of a bad surgeon will not have such doubts. After all, the maximum number of patients die on his operating table. Those who survived initially had a relatively mild illness and recovered quickly and easily. If a bad surgeon has good empathy, then thanks to the Dr. House effect, based on the results of a survey of patients, he can become the best specialist at the hospital.

The idea of ​​interviewing only surviving hospital patients is brilliant.
But no less daring is the idea of ​​interviewing only successfully graduated students of an online course. Unlike dead patients, students who dropped out of the course are probably still alive, so what’s stopping you from talking to them?

In a good course, out of 100 applicants, 90 students will reach the end. Among them there will be “severe” patients and “mild” patients. Is it difficult to organize a career track that will employ 90 people at once? Difficult.

In a bad course, out of 100 applicants, only 10 survivors will reach the end. Is it difficult to organize a career track where you only need to employ 10 people? No.

And moreover:

In any course, even the worst one, there are always 5% of students who will find a job anyway.

Thus, a bad course, where only 10 out of 100 people completed their studies, already has 5 people employed and the impressive statistics “50% of graduates are employed.”

Yandex.Practicum reveals on its landing page the number of students completing their studies. This is 60% (https://practicum.yandex.ru/jobreport). This is not HSE research data, but Yandex.Practicum’s own statistics. You can trust them, you can not – but at least it’s something.

Unfortunately, Skillbox does not disclose the number of people who complete the training. But based on the fact that in many respects the HSE research data on both platforms overlap, for the above calculations the number 60% was also taken using the expert method. In reality it may be different, but it is unlikely that it will be more than 60%.

“The number of people finding jobs out of the number of applicants is not our KPI”

Too often in conversations with course leaders, I hear the statement that the number of jobs should be assessed only by graduates, without taking into account those who completed their studies. Like, our job is to give people the opportunity to learn. And if a student came “just to look” and after studying decided not to look for a job, then this is not our problem.

As far as I see, this is the position of large platforms and specifically in Russia.

In Europe, especially in the Nordic countries, additional education for adults is indeed widely developed in the format “you need to constantly develop, follow trends, be comprehensively educated.” And there are really a lot of “I’m just looking” students here. But in Europe there is not such a big gap between IT and non-IT salaries, and people do not have much motivation in principle to switch to IT in the first place.

IT bootcamps are thriving in the USA, and this is a completely different story. Would anyone quit their current job and go to school full-time just for the extra social interaction or general self-development? Of course not. The student comes for a clear and specific goal.

Russian online education is trying to sit on two chairs. On the one hand, give purely applied courses like testing. Which, let’s be honest, are not the most exciting and to which hardly anyone will come “for fun”.
On the other hand, teach nutritional science, biohacking, partnerships And successful online dating. However, for some, the latest “specialties” can be a source of professional income.

In any case, it is obvious that trying to use the same metrics for both professional development and personal optionality is fundamentally wrong.

If a student came to study for the sake of development, then NPS, CSI and other parameters are quite acceptable.

If he came to get a new profession, then the main KPI is whether he found a job as a result or not. And in the case when a student was unable to complete his studies, then these are the problems of the course, not the student.

What should be a wake-up call for Skillbox here is that 21% of respondents had their learning goal change during the course of their training. This could be due to student procrastination. Or it could be because “with such quality of training I will never find a job.” But for some reason, as we said above, from the calculation of the percentage of those employed, those who, in principle, wanted to find a new job, but did not begin this search within six months from the end of the course, were excluded.

It is the number of students who reach the end of their studies that reflects the real quality of the educational platform. When teachers are trained to teach, materials are verified, and each student is given the help and support they need.

And only tracking the number of employed students, taking into account ALL initially admitted, and not just those who reached graduation, gives the course real feedback about the relevance of its educational program and how “in the market” it is.

conclusions

According to a HSE study, the Skillbox training platform demonstrates the statistics “26% of students initially admitted to the testing specialty get a job in IT.”

The data was calculated taking into account the percentage of students reaching the end of their studies, chosen by analogy with Yandex.Workshop, of 60%.
The HSE research methodology relies too much on the data provided by the course itself, does not take into account the percentage of students leaving their studies, and evaluates the initial motivation of students to find a job more than a year after the start of their studies. All this can influence the results of the study, distorting them for the better.

My expert assessment is that employment of the number of applicants is within 10-20%. The same Yandex.Practicum has a deep and resource-intensive course that requires the student to study up to 40 hours per week. But, according to another HSE study, its employment rate for Test Engineer graduates is only 39%. For Skillbox, let me remind you, it’s 66%. One possible explanation is that at Skillbox, less than 60% of applicants actually complete their studies. And this can greatly affect the resulting percentage of those initially admitted to finding a job.

But 26% is a very sobering number in any case. Because I still often see students who, after studying the course for six months, graduating and spending several months looking for work, end up saying in amazement, “less than half of the students in our group got a job.” Obviously, upon initial admission, they do not try to communicate with previous graduates of the course and buy into some marketing 93%.

Independent data

Unfortunately, my study did not have enough Skillbox students to provide data.

But within the framework of what I created manifesto of professional IT education Manifestus.Pro An in-depth audit of courses is performed that meets more stringent requirements:

– data on the composition of study groups at the time of admission and the time of graduation are checked independently, without relying on the information provided by the course;
– % of those employed is calculated from all students who entered the course, and not just from those who graduated;
– facts of employment of graduates are checked, and these statistics cannot be inflated by the course;
– all data collected as part of the audit is checked by an independent supervisory board, which includes managers of Russian and foreign IT companies, including Samokat, Yandex NV and others.

About how exactly the employment audit of a particular course is carried out and how the data is checked by an independent supervisory board – in the article next week.

Notifications about new articles appear in the telegram channel QAsmokeQA – Testing the quality of IT training.