Matching teachers and students with ML

I work as an analyst on the faculty recruitment team. We monitor the selection funnel from the moment the student pays for the classes to the selection of a teacher and conducting a lesson with him. We also deal with reselection – cases, when a student, having worked out with a teacher, decides to replace him. Our key metrics are average pickup time, the percentage of pickup automation (i.e. the percentage of pickups that didn’t require an operator), and metrics that characterize the quality of the pickup, such as the percentage of students who decide to change instructors within the first few lessons.

Therefore, the main task facing the selection department is to make sure that the student chooses the most suitable teacher for himself. This directly affects our core metrics. After all, if a student has chosen a teacher who does not suit him, then he will quickly change him or be completely disappointed and stop studying with us, or the teacher himself will refuse the student.

What does selection look like for a student? After paying for classes or requesting a reselection, the student is taken to a page with a list of teachers, where he can choose the desired number of lessons per week and get acquainted with the available teachers.

Since users are more likely to choose those who are closer to the top of the list, we are faced with the task of correctly ranking teachers, taking into account the characteristics of the student, teacher and subject.

old approach

Initially, we calculated a rating for each teacher, which took into account, with some weights, the characteristics of the teacher that we consider important. For example, these are the age of the teacher and the student, the size of the purchased package, various metrics by which we evaluate the quality of teaching, etc. This rating was updated every day and was used to rank the list of teachers. We could somehow change these weights and see how it affects the selection metrics in A / B tests, and thus gradually improve the model.

But this approach has several disadvantages:

  • The rating does not take into account the individual characteristics of students. We divided students into several large groups according to a number of parameters, but most of the individual parameters such as geography or language level were not taken into account in any way.

  • The rating is not universal for different subjects. The English Teacher Ranking was not well suited for math teachers and with the addition of new subjects, a new ranking had to be developed to suit the specifics of each product. With the advent of a large number of new products in the school, this approach became very inconvenient and resource-intensive.

New Approach

Since it is important for us that the student does not stop studying after several lessons, we can try to predict the number of lessons that he will spend with a particular teacher (hereinafter referred to as lifetime) using machine learning, based on the array of historical data that we have accumulated over several years. And use the predicted value for ranking.

There are quite a few parameters that affect lifetime. For example, let’s see how it correlates with teacher experience for different product lines.

For adult English experience does not correlate strongly with lifetime. For the rest, teachers with more experience keep students longer on average.

For English, it is also interesting to see how lifetime correlates with the level of English teacher and student.

Here the level of the student is measured on a 10-point scale, and the level of the teacher on the generally accepted European system. We see that for students with a weak level of English, teachers with a level of B1, B2 are preferred. However, as the level of the student increases, so does the preferred level of the teacher.

Let’s see how feature importance looks like for the most basic model, in which we have collected only the main parameters: teacher experience, geography, language level, desired number of lessons per week, package size purchased, etc.

Such a model gives some idea of ​​the parameters that affect the lifetime of the student, but still has a rather weak predictive power. To improve it, we had to include additional information that we know about teachers, for example: work experience, average lifetime of his students, average number of lessons per month, teaching quality metrics, etc.

You can read more details about the work of the model and future improvements in my telegram blog Good, bad, evil analyst.

Deployment

When looking for a solution for implementing the model in production, we proceeded from the following prerequisites:

  1. Since the calculations must be done in real time, the system must be fairly fast and reliable.

  2. The model should be isolated from the rest of the selection service so that you can quickly make changes and test new versions of the model without involving the development team.

  3. There should not be large infrastructure costs.

Initially, I considered a separate server with a model, which the selection service would access via API, or Amazon SageMaker (a cloud service for deploying ML models), which we sometimes use for ML tasks. However, since our model is quite simple and does not require large computing power, we ended up using the Skyeng infrastructure for containerized applications. We had a ready-made skeleton for a Python + Django application. On it, the API for our model was implemented.

Testing

Using our A/B testing platform, we divide students who need teacher matching into test and control groups and collect data on the performance of the new version of the model. The main metrics we look at are the number of lessons a student has in a fixed window and the proportion of students who reselect a teacher during the first few lessons. We also want not to drop the conversion in the choice of the teacher and not to increase the selection time.

Outcome

The new approach allows taking into account the individual features of the student and the teacher, as well as easily taking into account new products. And thanks to the isolated implementation of the model, we will be able to quickly change and test new versions of the model, with almost no involvement of developers.

Similar Posts

Leave a Reply