Shaking the foundations or from FAANG to an HFT startup

I am a server component developer with 10+ years of experience (C++, python). Not a trader, not a quant. This article is the result of my year-long observation when, after 8 years in large IT corporations, I suddenly dived into an HFT startup with a team of a hundred people. What surprised me the most about the difference in development approaches?
First, a few words about the parties in question:

• Pro FAANG, as an employer, everything is stated and known a million times. In the view of many developers, this is the last instance of career growth, free food, huge monitors, brilliant colleagues. I’m somewhat skeptical about these companies and have already had a bit of a whine here. The fact is that there is a lot of bureaucracy, long cycles and a common lack of interest in the result. But there is a good compensation and stabilityat least until recently.

• HFT (high frequency trading) – a niche of technology companies that are engaged in automatic trading on exchanges, and, based on the definition, they do this with a high frequency. The term is very non-specific, and the tasks, approaches, statuses in the markets of such companies are quite different. However, the technical solutions are similar – a critical factor for success in the field is the speed of reaction to market changes. Speed ​​allows you to bypass competitors and is well converted into money. There are a few big players in this area, but a lot of small companies are quite good ways to earn a comfortable existence (according to their hr).

In general, many of my observations can be explained by the importance of the speed of your programs and the frequency of release cycles. You might argue that speed is important in corporations as well. It’s not for nothing that they ask to expand all these trees at interviews and ask asymptotics. But what about savings on thousands of servers and megatons of CO2 emissions? But I disagree, because everywhere I worked, the more important properties of the systems were reliability and versatility, and then optimization, if you have nothing to do at all. Allocate 19 cores to a simple web service with a database at a load of 1 rps? Why not, but everything is reliable, replicated and protected from a detachment of excavators in search of data center cables.

So, I list the main theses and try not to violate any NDA:

The best solutions are simple

It seems to be nothing new and especially smart. “Everything ingenious is simple”, the Pareto principle, Occam’s Razor, etc. are in excellent agreement with this observation. But there is a nuance: many things around us are only outwardly very simple.

For example, you have a date + time in a text data stream, and they need to be parsed. Most developers will turn to the documentation of their favorite languages ​​for the right module, plug in this thing, and be good 99% of the time. One call to convert the text into the desired structure and you’re done, what could be simpler? But if we look at what happens in this module, we will find work with the current OS locale, some incomprehensible virtual functions for working with the calendar, all kinds of date formats, and who knows what else. And you always have the same format, the data is always in the same order, and then it turns out that you only need the month and day from the date. Code that does exactly what we want it to do will obviously run faster.

Of course, you won’t get far with one such improvement, but the principle emerges – minimalistic, often self-written solutions provide the best efficiency. This approach is more difficult to deal with and needs to be more actively supported, which is often redundant for large companies, but, on the other hand, in a very highly competitive environment it is justified and turns into money. Therefore, there are a lot of hot paths in HFT, and you still need to be able to do a simple one with high quality.

More bikes for the bike god

Using ready-made solutions is the cornerstone of effective development, but the previous point leads us to the fact that many things will have to be written independently. And not only because their speed is low, but also because you do not trust third-party solutions. And the price of a software error (or hack) can be too high. For example, the disappearance of your company.

Although I note that bicycle solutions, which are out of place in decent places, are just as natural in very large corporations. In many FAANG companies you will find custom solutions for common tasks. There, this is most often motivated by high workloads, the desire for independence from third-party support, as well as the presence of unemployed developers.

Lack of authority for standards and approaches

Previously, I rather naively believed that the ability to write highly loaded systems in C ++ somehow intersects with multithreading. He actively used it in his projects, studied the properties of the memory model, the semantics of atomic types, etc. And then, in an environment with a decent load and huge importance of delay, I met the postulate of “no multithreading”. Because it’s complex, it makes the code non-obvious and has a known overhead. And overhead, as we have already noticed, is bad. And here I remind you that some thread synchronization also sprouted into such harmless C ++ tools as static and into some smart pointers. And here you are in 2023 no longer embarrassed by the use of raw pointers that you have been avoiding for many years. After all, you know what you’re doing. What can go wrong?

Faced with situations like this, you realize that many teachings about good coding or design cease to be authoritative in extraordinary situations. Actually, they were never intended for this.

I also note that this applies not only to code – popular infrastructure solutions, of course, are also being revised, be it virtualization/containerization systems, transport protocols, etc. Infrastructure for high-frequency trading is as important as code.

Availability is not so important

Working in huge companies, one day you may find yourself in a series of hour-long meetings where scenarios that your service may face are discussed. And these scenarios are suitable for some kind of dystopia. Because they are all very sad, but completely impossible.

For such companies, reputational losses from downtime are very bad, investors do not like this. So they want their company’s services to be up and running 100% of the time. But developers spend a lot of time developing and supporting such projects, protecting against impossible situations and other ways of insurance, which they could spend on launching something new. Something that will bring in more money than downtime protection for a couple of hours every few months. Although, the absence of this protection will make your work a little more nervous. Especially while on duty.

In general, the idea of ​​insurance only where “thin” and the redundancy of one hundred percent availability may surprise you after the corporate stability mania. By the way, the same applies to high percentages of test coverage.

Naim died

More precisely, it is severely broken in corporations. Algorithmic sections that test diligence in solving the top 100 litcode, behavioral and “cultural” sections that test your ability to embellish, this is all very interesting and, probably, can give objectivity in a large stream of people. However, this approach obviously does not test the knowledge and skills that your team needs directly. And in places with a bootcamp, getting a new employee into a random (relative to his skills) team only exacerbates the problem. As a result, it turns out that the level of colleagues, after such filtering, is not as high as it is commonly believed.

In companies where hiring is not put on the pipeline for six months in advance, but which are also able to take the best from the market, your future team will evaluate you on really important skills, which means that the chance of random people around will be lower.

Whether this is a problem or an advantage for you is an open question.

Interesting daily tasks

Not everyone likes to work “in finance”, and some of the arguments against are quite logical. A modest impact on society, for example. Megacorporations, on the other hand, position themselves as important phenomena of our time, which here do not just earn money from advertising, but sow reasonable, kind, eternal.

True, while being there and optimizing some metric that no one will ever notice an increase anywhere, you can also rightly ask yourself: do not nonsense do i do? On some projects, you will generally ignore the code for months, write documents and attend endless rallies. And only in words your work will seem to others as something cool.
Most HFT companies don’t find it very profitable to grow, and if you work for a company with a hundred times less staff, you will be a cog in a hundred times more. And even if the impact on the world around you is not as big as you would like, the likelihood of engaging in interesting engineering tasks can be significantly higher.

What is the result

All this can be summed up as the rejection of generalizations and the reduction of extravagance. This seems logical, as large companies want the versatility to reduce the threshold for entry into their projects, and are also willing to pull some cost overruns. In HFT, wastefulness directly hits your annual bonus, and small teams of experienced specialists can afford a high barrier to entry. These companies want your code to help make money, not just cool abstractions. And while this applies to all commercial development, it feels more acute there, and much of what is considered good form everywhere can be questioned in favor of instant profit.

On the other hand, the described practices may well make you very “deformed” and not very suitable for work in other places. I would recommend moving from the general to the specific. But it’s probably not worth dreaming only of huge corporations as a place where you can sit until retirement.