Fintech Development or How to Go Through the 7 Circles of Hell to Bring a Product to Production

Hello everyone! Let me introduce myself: I am the head of the development team and part-time PM, coming from the world of systems engineering and hard analytics, more details in the channel The Inside of IT.

The purpose of this article is to tell the real problems that a team faces when creating a new product and its leader in a large fintech and yes, not just from someone's words, but having felt everything on my own skin. Last year, I took on a fairly large and serious project, where it was necessary at the start:

1. assemble a team from scratch, develop the necessary profiles of employees according to competencies for the project;

2. lay down the architecture of the information system, solution and project as a whole, which should not change radically later, and what is even more important – highly loaded with all the nuances of reliability, be prepared not just for some couple of hundred RPS at the entrance, but to expect an order of magnitude more in the flow;

3. reengineer an existing business process and rework a ton of batch solution code;

4. prepare the team for the corporate production process;

5. to work with a completely new team and move from storming to performing (not easy at all).

When I am faced with tasks, be it creating a data management system, automating document flow using artificial intelligence functions, creating a solution for high-performance data processing in real time or simply implementing software for business, I am ready for action, but starting everything from scratch and only with a business need is a new challenge! Previously, I created from scratch and designed graph solutions, data management systems, supported and developed a geo and ML platform, so my experience in development and design is far from one year and in completely different companies. In my coordinate system, any client's need is not just a task, but an opportunity to realize the potential of technologies for the benefit of business, so I am ready to evaluate any business request from financial costs and budget to the necessary resources. Each project is a complex equation where the factors of time, clarity of reproduction of requirements, finances, speed and resources are balanced taking into account the risk coefficient of the project. However, in the world of fintech, the rules are changing, especially in the current realities, when everything is on an import-substituted stack, and the requirements for flexibility and adaptability are multiplied and sometimes many times over, and the risk coefficient is increased in advance. IT development in the financial sector and in the world of large corporations is not just about completing tasks, it is a constant passage through the circles of hell (maybe this is my feeling), where you need to be able to respond and be ready to respond to challenges that can come from anywhere.

In general, I want to share what can await the head of the development team (PM, PO) if he starts doing a project in a large enterprise, I will describe fintech. Perhaps this will not be on your path, then you are a happy person))

Circle 1 – Long Term Recruitment

Now to the problems – long and exhausting hiring is probably one of the most difficult and important steps, because the success of the entire project depends on the future team. And the hired employees must be qualified, have similar or relevant experience, and given the tight deadlines – this is a challenge.

I have already said that hiring qualified IT specialists is difficult and complicated, and especially now, well, you know what I mean, now there is a real shortage of highly qualified personnel. And in our environment, where competition for good personnel is especially intense, hiring becomes quite exhausting, add a couple more pinches of bureaucracy – mmm, what a dish)))).

From creating job posting requirements and posting ads to conducting multiple rounds of interviews and screening candidates for security compliance, each step of the process requires attention, time, and more time. Moreover, in the world of fintech, where privacy and data security are of the utmost importance, the screening process can be especially thorough and lengthy; of course, they won’t sit here for regular positions with lie detectors, but nevertheless. The team leader has to literally fight his way through the recruiting maze to find those specialists who not only meet the project’s requirements, but are also ready to invest their energy and knowledge in achieving its goals, and also make sure that this person is the one. This requires not only professionalism and experience, but also patience, persistence, and the ability to make decisions in conditions of uncertainty and non-standard situations.

I have some friends who are managers who spend months looking for specialists, really, guys??? For me personally, you can almost always find a specialist, quickly enough, even if a corporate hunter can't help, I go to my blog, articles, messengers, conferences, channels, communities, friends and find the right guys, this is not just “bragging”, but facts. Therefore, in order for the search for a specialist not to be in an emergency mode – plan your project and the development process. And if a problem has arisen, do not hesitate to use different approaches to finding candidates and remember! Hunters with their so-called “speed of light” are not the limit and be proactive.

When I had the task of urgently assembling a highly qualified development team, I first formed a portrait of each candidate, a stack and preliminary architecture, a hiring priority, and then went looking for people everywhere. If you and I know each other, I think you have seen my posts more than once. As a result, in 2 months the team was assembled, where corporate HR managers helped with only two. I spoke in more detail in the reports at the HighLoad++ 2024 and Project Analysis and Management 2024 conferences.

Circle 2 – Bureaucracy and tons of paperwork

As you understand, security and compliance with regulatory requirements come first, so every change in the system must be strictly documented, approved by the relevant departments, services and management bodies. This means that in order to simply start working on a project, you have to go through a whole series of formal procedures and fill out a bunch of papers, only not at the physical level, but virtually, but this does not diminish the complexity of the procedures and processes. Next comes the preparation of documentation on requirements (BT, FT, NFT), architecture, testing and many other aspects of the project as part of the development process, each of which must be described in detail and accurately, as well as justified. Time is “devoured” here for the entire team, especially for analysts and testers, and the team leader is required to be able to work under conditions of increased responsibility and stress, as well as effectively coordinate the work of the entire team so that all documents are completed on time and comply with corporate standards and rules.

If you go to large companies, then be prepared for a lot of documentation, I'm not kidding – a ton of documentation… I remember when I worked in one of the companies in the manufacturing sector, we only needed a technical assignment and a description of how the client could use our application – that's all, and now one step to the left/right – a package of agreed documentation.

If you go to large corporations, expect a lot of documentation (it all depends on the company) and that it will constantly eat up time during the development process and take away the team's resources.

Circle 3 – Customer with constantly changing requirements

The team has been assembled, the documentation has almost been overcome – success?? But what about the new requirements? Let's slightly correct the specification of several API methods, and change the business process a little, well, replace a couple of actions there ))) For business, everything is always simple, but when you start researching and analyzing, yes, you understand that, as if after a minor edit, development in a couple of man-days, functional and regression testing …. documentation ….

Working with a customer and business as a whole, which constantly changes its requirements and indications, as you understood, this is the next hellish circle before the release. In the conditions of constant changes everywhere and everywhere, as well as the formats of planning projects, products and their control points, we willy-nilly come to the fact that the requirements are constantly changing and sometimes dramatically, and what does a sharp change mean for the well-developed architecture of an extremely complex process? Frankenstein, inconsistency of requirements and bugs … .

Yes, businesses still revise their plans and strategies, but they very rarely have checkpoints, which can lead to serious delays on the part of IT, although IT is not to blame at all – they are victims in fact. The one who leads the team management process is required to be flexible, adaptable, able to quickly respond to changes and reconfigure the team and development processes, because it is necessary to meet new requirements and expectations. Fortunately, most of the IT world now lives according to “scrum” and constant changes are the norm (I will not go into deep reasoning here, because everyone has different points of view). In addition, the lead, like a tennis player, must manage to parry serves on time, be able to convince the customer of the need for certain changes and assess their impact on the project deadlines and budget in order to ensure its successful implementation in conditions of constant uncertainty and instability. And the development team must adapt to reality and, in case of severe overloads, be able to convey the team leader about the problems.

In general, there will always be changes, be prepared for this, here you just need to clearly show the business that any change in requirements is a change in deadlines and if you want to hit the project checkpoint, then be prepared to increase the error of hitting the business need, since due to some requirements you have to change the code several times, and sometimes even rewrite it completely. Having a project route map on hand will make your life easier.

Circle 4 – Lack of resources and complexity

IT projects often require significant investments in both equipment and technology, as well as personnel and development processes. Do not forget that the software development process involves not only the creation of code, but also a comprehensive testing process. In the world of financial technologies, where reliability and data security are of the highest priority, this process becomes especially important and labor-intensive, and instead of two contours you have 4, 5, 6 … and then it depends on corporate approaches, rules, etc. Perhaps you have heard of such development contours as:

• Development and Experimentation Loop – designed to create and test new product features and capabilities. Developers here are free to experiment and explore new ideas to improve the user experience and functionality of the product.

• System and functional testing loop – testing of various aspects of the product, including its functionality, performance, security, and compatibility with other systems. Sometimes regression testing is also carried out here, but in general, work is done here to identify and eliminate errors and defects to ensure stable and reliable operation of the product before its integration testing.

• The integration testing circuit is testing the interaction of the product with other systems and services with which it must interact.
  Load testing circuit – testing the performance and scalability of the product under various loads and usage conditions, checking how the product works under high loads, trying to optimize its performance and efficiency.

• Pre-production and production loops — designed to prepare the product for release and its implementation in the production environment. Here, final tests and checks are carried out, and support and maintenance of the product are provided after its implementation.

  A company cannot always allocate enough resources to implement a project, especially if it involves the use of new technologies or the development of complex systems that require a lot of hardware and more than one stand. Therefore, it is necessary to look for compromises and alternative solutions to ensure the implementation of the project within a limited budget and time frame. So be prepared to optimize costs and use resources efficiently, and be prepared to take risks and non-standard decisions in conditions of uncertainty and limited opportunities.

Circle 5 – Conflicts with colleagues

The more interaction with different departments and teams, where everyone has their own interests and priorities, the more difficult it is, and there are many opportunities to come into conflict. Just imagine that team “A” is developing a corporate portal and has an extremely tight deadline, and then team “B” comes in and asks to make an API for them, and then the question arises, when? Therefore, the interests between the teams do not always coincide, everyone has plans and goals for the outlook of their project or product, and the larger the corporation, the more difficult it is, even though the company is one for the company. Therefore, the IT leader has to be not only a technical expert, but also a leader, a mediator who knows how to find a common language with different parties, find compromises and solutions that satisfy all participants in the process.

Circle 6 – Integration into the pipeline

As I mentioned earlier, the work of the development team includes not only pure development (see above))), but also important aspects such as implementation in the pipeline or automation of deployment processes to the production contour. It is important to understand that new functionality may be blocked pending the creation or configuration of the pipeline, possibly for more than one sprint, especially if your project is just starting out. Preparing the pipeline and testing it can take weeks, especially in the case of multiple contours from DEV to production. Our team was recently the first in the company to go to production on a completely new pipeline, and the process of debugging and configuring it turned out to be extremely complex and resource-intensive. If development took X amount of time, then integrating and configuring the CI/CD pipeline took about 20%. Something like that))

7 — Support and Maintenance

You see, the development team has limited access to senior contours, such as pre-production and production. This means that they do not have the ability to make changes and release releases on their own. For this, there is a specialized department – technical support. Surely you think that professionals work in TS? However, alas, most often it is staffed by employees whose skills are significantly lower than those of developers. And this is where the problems begin.

Support often does not fully understand what is happening in the system. In addition, one technical support employee is usually responsible for several information systems. This leads to a situation where he is not able to deeply understand each system at the developer level, which is often unrealistic. As a result, the second problem arises: support overload. But the third problem is a classic story of any corporation: bureaucracy, I have talked about it and will touch on it again, because in order to solve even a small problem, you will have to make a justification and fill out a bunch of documents, descriptions, plans ….

From my experience of working with support, I can say one thing: strive for mutual understanding! It is necessary to find a common language and come to an agreement. I had a case when one of my managers quarreled with support, and this led to serious consequences: mutual accusations, more careful control over each step in development – it was like hell, especially considering the importance of the project. Now I try to find a common language with support as much as possible, otherwise you yourself understand what can happen!

Conclusion

Heads of development teams (PO, PM) in fintech, especially if these are large financial organizations, face many challenges and obstacles on the way to successful implementation of projects and sometimes there is a feeling that everything is against you creating something. But with the right approach, flexibility, adaptability to processes and the ability to make informed decisions, you can overcome any difficulties and ensure the achievement of your goals. Why do large IT specialists work in fintech and do not go to others, where there are many times fewer hellish circles or they are almost absent? Finance, stability, an interesting project, technology, a team, and then whatever you have. The main thing I want to convey is that if you are afraid of all this, then do not even go to such companies, but if you are not afraid, then this type of company can help you in serious growth, provided that you, like an electron, do not stand still and take a proactive position. I hope the article turned out to be interesting and useful. See you in the next articles, conferences and channel The Inside of IT.

P.S. Thanks to my team – you are awesome, without you none of this would have worked!