How to grow from two dozen to 700+ virtual machines in the cloud and learn to withstand powerful loads: the BelkaCar experience

Not so long ago, on our blog, we shared the story of an emergency migration of a client’s infrastructure. However, there are situations when a company does not need to migrate from anywhere – it immediately comes to “live” in the cloud without the experience of using on-premise under its belt. This was the case with BelkaCar – the startup immediately deployed services in our

public cloud

… Let’s discuss how correct this decision was and how the company plans to develop its infrastructure.

Today BelkaCar is one of the largest car-sharing services in Russia and one of the ten largest car-sharing companies in the world. BelkaCar operates in Moscow and the Krasnodar Territory, and the total number of users has exceeded 2 million, and about 700-750 thousand of them use the service every month.

But this is now. When car sharing first approached us 5 years ago with the intention of renting cloud resources, it was a typical startup not associated with any major project. Like any such startup, BelkaCar had to exist for some time in conditions of unpredictability and uncertainty. It was not known whether he would shoot, and if he did, then when.

But even in such a situation, it must be borne in mind that one day the number of users and the demand for services may change. The business will be delighted, and the CIO and technical staff must be confident that computing resources can withstand the multiplier load. Therefore, BelkaCar immediately chose the cloud: IaaS allows you to quickly scale (at least horizontally), not reserve additional capacities and, which is fundamentally important for a startup, not to increase CAPEX, working only with optimal OPEX costs.

Infrastructure: then and now

Having decided on the provider, BelkaCar rented a couple of dozen virtual machines in the cloud. The first services were launched that provide the main functionality of car sharing – services for interacting with cars (telematics), services for managing users, trips, tariffs, billing services, basic internal administrative services for managing the fleet.

5 years have passed since then. The number of users grew from a few hundred to almost two million, and virtual machines began to number in the hundreds.

No matter how simple the work of the service may look for the user, in fact, car sharing is quite complex, both organizationally and technologically.

Now the infrastructure of a startup, united into a single car-sharing platform, consists of a large number of services:

  • mobile applications for individuals and corporate clients;
  • telematic services;
  • fleet management (maintenance and repair, refueling, car washes, tire fitting, work with road accidents, car hauls);
  • CRM, call center and other services of the customer service department;
  • user verification;
  • debt collection;
  • dynamic pricing;
  • billing with PCI DSS Level 1 certification;
  • communication with users (reengagement);
  • behavioral analytics;
  • safe driving;
  • numerous integrations with external services.

In addition, new business areas began to actively develop in 2021. A car subscription platform was launched, and the first subscription services for ŠKODA and AUDI cars built on it were launched.

To ensure the required level of reliability and performance of services, the company uses more than 700 virtual machines for prod, stage and test environments. The total amount of consumed resources fluctuates at the level of 3000 vCPU, 3 Tb RAM and 300 Tb storage.

The infrastructure of carsharing exists in a continuously highly loaded state. Peak loads exceed 4000-5000 requests per second. The main contribution to the load profile comes from the telematics service, which connects the servers directly to the vehicles. It is he who is responsible for ensuring that users can rent a car in automatic mode – open / close it, start / stop the engine, warm up the interior remotely. The service also receives from the car a large amount of information about the coordinates of the car on the map, the state of its nodes, the occurrence of situations typical for road accidents, as well as about the user’s behavior while driving and his driving style.

Where the cloud shows its best

Flexible and efficient scaling

From time to time, BelkaCar, like any other company, is faced with peak infrastructure loads during marketing activities. For example, the conversion of bulk push notifications to application opening sometimes reaches 20%, and then the load on certain critical services may briefly exceed the typical one by 4-5 times. A short-term increase in resources on virtual machines of specific services helps to survive such situations without crashes and serious slowdowns in the platform’s functionality.

But there are also the opposite situations. In 2020, car sharing services faced a covid ban on working in Moscow, which lasted from April 13 to May 25. Then BelkaCar “curtailed” cloud resources almost several times, and along with them, hosting costs fell – this helped to survive, albeit a short, but nevertheless difficult period for the company.

Machine Learning and Data Analysis

BelkaCar is actively engaged in the processing of the collected data, in particular, in such areas as:

  • dynamic pricing based on various factors – the congestion of the vehicle fleet in certain zones, predictions about the direction of movement of the user, weather factors;
  • predictive analytics on the state of car units in the dynamics of their operation, in which partners are interested, primarily car manufacturers;
  • safe driving and user scoring;
  • modeling for calculating the start / end of the lease geofences;
  • analytics for search and localization of fraud-activities of users;
  • testing product hypotheses based on accumulated data sets.

All these activities require a short-term allocation of a large amount of resources, which is still easier to implement in the cloud than on your own site. The company takes a certain amount of power (vCPU, RAM, etc.), and when the task is completed, it releases them and stops paying.

Protection of personal information

Personal data protection is an extremely important and highest priority task for companies processing large amounts of personal data.

The current legislation of the Russian Federation is replete with regulations of various levels that provide regulation of this sensitive area. Every year the range of these acts covers more and more new areas, the level of requirements for the security of ISPD is increasing. Any company that processes personal data of Russians must meet the requirements, the main of which are indicated in:

  • federal laws 152-FZ, 242-FZ;
  • FSTEC order No. 21 and their methodological documents;
  • Resolution of the Government of the Russian Federation No. 1119.

Accommodation in

secure cloud

removes a huge part of the work to bring the infrastructure in line with the law. The use of cloud segments, protected in accordance with the requirements of 152-FZ and localized on the territory of the Russian Federation in accordance with the requirements of 242-FZ, helps to protect data at the technical level and greatly help in the successful passage of regulatory audits.

PCI DSS 3.2 Level 1 Certification

Car sharing processes a large number of financial transactions (over 10 million per year), which, in turn, leads to the need for certification in accordance with the PCI DSS Level 1 standard (with the number of transactions over 6 million per year). The fact that we have a certificate of conformity with the requirements allowed BelkaCar to significantly reduce the number of prepared documents and technical measures for certification and successfully pass certification.

Backup

Alas, the importance of tiered and geographically dispersed backups is often underestimated when choosing cloud technologies. Loss of data can negatively impact your business.

BelkaCar uses several approaches to backup:

The ability to build hybrid solutions

Companies, which are constantly increasing the load on services and the volume of these services in terms of data, at some point think about creating a hybrid cloud.

Then the company decided to follow the already proven path and also use the provider’s capacities. Now BelkaCar, together with our team, is working on building and migrating resources to a dedicated secure cloud. At the same time, critical services will be placed in exclusive cloud resources, in parallel with which the expandable resources of the public cloud will be used. Such an architecture, on the one hand, guarantees isolation from other consumers and allows you to be confident in control over the main resources of this segment, and on the other hand, it makes it possible to use additional capacities in cases where they are needed.

Similar Posts

Leave a Reply