experience of Reg.ru and DigiLabs
Text navigation:
→ How we calculated the performance index
Where it all began
It all started with our Reg.ru development team, in collaboration with DigiLabs, developing a platform for 1C, taking into account the specifics and requirements of the application. And then we needed to prove first of all to ourselves (and then to our clients) that the infrastructure is really suitable for hosting 1C. Daria Velichaeva's team (DigiLabs) prepared a testing methodology with a full imitation of intensive user actions.
Spoiler: the test showed that our platform is at least 30% more productive than other clouds currently available. With some providers, the difference was up to 90%.
There is a well-known and widely used load test, but it has a drawback: single-threading. Therefore, it shows not quite relevant results. Another drawback of the existing tests, in our opinion, is the minimum amount of data in the test databases, since owners of databases of significant volumes often encounter performance problems. Having failed to find a ready-made method for objectively comparing infrastructure performance, we decided to develop our own test that allows us to more accurately assess the performance of multi-user databases.
So, the task was this: to check how well the platform we assembled could handle the high load from 1C.
What we did
We took one of the most resource-intensive configurations — “1C:ERP Enterprise Management”. We filled it with a sufficient amount of data (150 GB, and then also conducted a test on a 500 GB base). Using the “Test Center” subsystem, which is part of the “1C:Corporate Toolkit”, we developed a multi-user scenario that emulates the work of users. Each virtual user in the multi-user test performs a sequence of operations that simulate the sales process. The test is limited in time (15 minutes), the number of operations performed depends on the performance of the stand.
When developing the testing methodology, we tried to make a comprehensive assessment of performance, so in addition to the speed of operations, we took into account the degree of load on the 1C server and DBMS server equipment. The following indicators are analyzed during the test:
number of key operations performed;
average, maximum and minimum execution time of key operations;
APDEX integrated performance assessment;
utilization of the 1C server processor and DBMS server;
queue to the disks of the 1C server and the DBMS server;
1C server and DBMS server disk latency;
memory consumption of the 1C server and the DBMS server.
Thus, even having received similar values of the performance assessment of different stands in terms of execution of operations, we can see where the hardware is overloaded and where it is idle.
To collect OS metrics, we use the telegraf service; after the test is complete, we upload the metrics to 1C.
To conduct load tests, we selected the following server configurations:
DBMS server: Windows OS, 48 vCPU, 128 GB RAM;
1C server: Windows OS, 32 vCPU, 64 GB RAM;
1C platform version: 8.3.23.1912;
MSSQL version: Microsoft SQL Server 2019 (RTM) – 15.0.2000.5 (X64) Enterprise Evaluation (64-bit);
VMware ESXi: 8.0.2, 22380479.
To reproduce the load, 100 VRMs (virtual workstations) were used. It is worth mentioning here that this number does not reflect the number of real users who can work in this server configuration. VRMs create a more intensive load. The purpose of the test is to compare different stands with the same resources and the same load.
In addition to the multi-user scenario, another one was prepared: with the “pain” of many – the routine operation of closing the month. It is long and complex, but very important for users.
How we calculated the performance index
In the process of testing and discussing the results, we came to the conclusion that the business asks for simpler and more understandable metrics. Therefore, we left such concepts as the overall integral assessment, quantiles of the OS metrics and APDEX for detailed analysis, and as the most visual and understandable results we used the number of operations performed in the allotted time, the average time of execution of key operations, and we also did not forget to compare the cost of the infrastructure, so we introduced such a metric as the number of operations per 1000 rubles.
We compared the average execution time of the main key operations of a multi-user scenario:
An important factor when choosing a cloud provider is cost, so we compared the cost of the tested configurations:
When conducting the second test, which included closing the period, the performance dynamics were maintained, we received similar figures for the increase in the performance of our infrastructure compared to others.
One of our customers has done their own load testing of different platforms using other methodologies, including HammerDB. And their test confirmed our performance numbers.
What's the bottom line?
Now the entire Runiti group has transferred part of its databases to the infrastructure developed and tested jointly with DigiLabs (and this is the main indicator of our trust in our product). The platform suits us both as an internal customer and as a service provider: our clients also use this product.
