Why do so many data scientists quit their jobs?

To the start of the flagship data science course share the opinion of an ML engineer about what frustrations you can expect at work as a data scientist, so that you are ready for them. For details, we invite under cat.

Yes, I am a data scientist, and yes, you read the title of the article correctly, but someone had to say this. There is a lot of writing these days about how data science is the coolest work of the 21st century. This area is full of highly qualified specialists capable of solving complex problems. The smart ones have turned into data scientists with huge earnings, and now it’s just a dream job.

But the truth is that (as they say in this Financial Times article) data scientists typically “spend 1-2 hours a week looking for a new job.” The article also states that “machine learning specialists topped the list of developers who said they were looking for a new job (14.3% of them). In second place are data scientists (13.2%).” This is a survey of 64,000 developers on Stack Overflow. I’ve been in that position myself and recently retired from data science.

Why are data scientists so eager to find a new job?

Before answering this question, I must clarify: I continue to be a data scientist. In general, I enjoy data science and don’t want to discourage others from becoming data scientists – this job can be fun and rewarding. I just wanted to be the devil’s advocate and show the negative aspects of this job in the article. I think there are 4 main reasons why many data scientists are dissatisfied with their work.

1. Expectations do not match reality

“Big data is [то, о чём все говорят], no one really knows how to do it, but everyone thinks that everyone around them does it, and therefore everyone says they do it … “- Dan Ariely.

This quote is straight to the point. Many data science newbies (myself included) would love to do it, solving complex problems with cool new machine learning algorithms that have a huge business impact. It seemed that this work is the most important thing that we have done before. But usually this is not the case. It is difficult to compile an exhaustive list of reasons for the exodus of data scientists from this area, but the main one is inconsistency of expectations with reality.

Every company is different, but many of them hire data scientists without having the right infrastructureto start building something of value out of AI. It makes it worse problem its cold start. In addition, these companies do not have experienced specialists to help newcomers – this is a recipe for disappointment on both sides.

A data scientist came to the company to write smart machine learning algorithms to make business decisions. But he cannot do this, because his primary task is to understand the data infrastructure and / or create analytical reports. The company needs a chart to be shown at a board meeting. Then frustration sets in: the board doesn’t see results fast enough, and the data scientist is dissatisfied with the job.

Robert Chang provided a very helpful quote., Giving advice For beginner data scientists:

“It is important to evaluate how well our aspirations fit into the main direction of our work. Find projects, teams and companies whose focus best suits yours.”

This underlines the two-way nature of the relationship between the employer and the data scientist. If the company goes in the wrong direction or its goals don’t align with the goals of the data scientist, sooner or later they will leave.

For those who are interested: Samson Hu there is a fantastic series of articles on how the team analysts at Wish.

Another reason for the disappointment of data scientists is similar to the reason my disappointments at Academia: I thought I could really influence people everywhere, not just in the company. But if the company’s core business is not machine learning (my previous employer was a publishing firm), then it is likely that data science will bring only small additional benefits. Of course, you may be lucky and something very significant awaits you, or in some project you will stumble upon a “gold mine”, but this happens less often.

2. Politics rules the show

There is already a brilliant article on this topic, dedicated to the issue of politics: The most difficult thing in data science: politics (“The hardest thing about data science: politics”) and I highly recommend reading it. In its first lines – what I want to say:

“When I woke up at 6 in the morning to learn support vector machines, I thought: “This is really hard! But I will be very valuable in the eyes of the future employer! If there was Time Machine, I would go back in time and popularly explain what I was wrong about.

Do you seriously think that knowing a lot of machine learning algorithms will become the most valuable data scientist? Then go back to the first point: expectations do not match reality.

In reality, the people with the most influence in business should have a good opinion of you. That is, you will have to constantly do ad hoc work, such as getting numbers from a database and giving them to the right people at the right time, doing simple projects just to ensure that the right people have the right idea about you. I often had to do this in the previous place. As annoying as it is, it’s a necessary part of the job.

3. Everything related to data is contacted by you

Those same influencers often don’t understand what a data scientist is. That is, you will be an expert analyst, and a specialist in reports, and an expert in databases.

Moreover, not only managers will reveal your new skills, but also colleagues who believe that you know everything related to data. You know Spark, Hadoop, Hive, Pig, SQL, Neo4J, MySQL, Python, R, Scala, Tensorflow, A/B Testing, NLP, all machine learning and everything related to data.

By the way, if you see all this in job descriptions, run without looking back. Companies that have no idea about data strategy have such ones. They will hire anyone, believing that they will solve all data problems by hiring anyone who is in any way connected with data. But that’s not all. Since you know all this, then, of course, you have access to all the data, and answers to all questions … and as quickly as possible.

Explain to everyone what you really are deed know and control can be difficult. Not because it will somehow affect the way you are treated, but because as a novice data scientist with little experience, you will be worried that people will think less of you. This can be quite a difficult situation.

4. Work in an isolated team

Successful data products are usually well-designed, intelligent user interfaces and, most importantly, a useful outcome that is at least perceived by users as a solution to the corresponding problem.

If a data scientist spends time only learning how to write and execute machine learning algorithms, then he may be only a small (albeit necessary) part of a team that leads a project to create a valuable product to success. These isolated teams will struggle to create something of value!

However, many companies have data science teams that develop their own projects and write code in an attempt to solve a problem. In some cases this is sufficient. For example, if you need a static spreadsheet once a quarter.

On the other hand, if the goal is to optimize the provision of intelligent suggestions in a custom website product, then this will require a lot of different skills that most data scientists do not have (only a real data scientist can solve this task).

Therefore, if a team working in isolation takes on a project, it is unlikely that it will succeed (or take a very long time, because in major companies to organize the work of isolated teams on a joint project is not easy).

Therefore, to be an effective data scientist in your field, it is not enough just to prove yourself in Kaggle contests and take online courses. Unfortunately or fortunately (depending on how you look at it), it depends on understanding how hierarchies and politics work in business.

Finding a company with a focus that matches your aspirations is a key goal when looking for a job in data science. And you may also need to adjust your expectations for future work. I hope I didn’t discourage you from becoming a data scientist. Thank you for your attention.

And if nothing scared you, we invite you to our Data Science or Python courses:

Learn more here.

Other professions and courses

Data Science and Machine Learning

Python, web development

Mobile development

Java and C#

From basics to depth

As well as

Similar Posts

Leave a Reply