“Let there be light!”, Or How we changed the UPS system in the data center in the midst of a pandemic

Upgrading a data center during the COVID-19 pandemic is not an easy task, especially when it comes to upgrading the site’s power supply system. Nevertheless, with a clear plan of action, a reliable contractor and a competent project team, it can be dealt with even in conditions of force majeure.

We will tell you how everything went and how our site in St. Petersburg became even more reliable for customers in terms of fault tolerance.

Delta UPS in Linxdatacenter Data Center

Introductory conditions

The uninterruptible power supply system (UPS) of our data center in St. Petersburg was originally designed according to the 2N model.

This means that if one UPS fails (or is shut down for maintenance), the other takes over the full load. Accordingly, the actually used power of one uninterruptible power supply (UPS) in a pair should not exceed 50% of the maximum available. This ensures the reliability of the system.

In normal operation, the UPSs provide the required 100% power, sharing the load equally.

However, the growth of loads on the data center led to the need for modernization: the level of capacity consumption was approaching the maximum permissible threshold.

Due to the high demand for the site’s resources, we made a decision in favor of modular UPSs as a format that optimally meets the challenges of data center development at the current moment and allows increasing capacity in the future.

At the same time, it turned out to be impossible to implement modular UPSs in the UPS built on monoblock solutions. It would be necessary to disconnect some of the equipment, which would lead to downtime of the clients’ IT systems, so we discarded this scenario right away.

It was necessary to create a fundamentally new UPS, redistributing the load between the existing monoblock UPS and new modular ones. At the same time, it was important to comply with the Uptime Institute requirements in terms of monitoring loads – up to 50% of the maximum per UPS unit.

Choice in favor

The choice fell on the equipment of Delta Electronics: the required UPS model Delta DPH 500 kVA was available in a warehouse in St. Petersburg, and the solution integrator (Tempesto Group of Companies) also had the status of a mono-brand distributor of the vendor in Russia, which played a big role for us along the way. project.

The modular format of the UPS allows you to solve many tasks related to the power supply of the data center, the main one of which is to increase fault tolerance.

The fact is that monoblock UPSs fail entirely, triggering a “domino effect” along the entire chain of SLA fulfillment. In contrast to them, modular UPSs in the event of an emergency “fly out” modularly, losing 50 kW each, which, with proper configuration of the UPS architecture, makes it possible not to feel the consequences of such failures, and in some cases the effect of such an accident will tend to zero.

In addition, modular solutions allow for the replacement of faulty modules “hot”, are easily scalable to fluctuating loads on the computer room, and are much more compact (a modular UPS occupies one rack versus 2.5 racks for a monoblock device). The economic factor is also important: modular solutions cost almost half the price compared to the market-leading monoblock models.

In short, it was a pretty easy choice.

To work

The project team consisted of 15 people. The team consisted of ten installers, a chief engineer, as well as his deputies in various areas (automation, electrical, etc.).

Work began in April 2020 with an eye to completion by June. However, it was not so easy to follow the plan: the task was complicated by the COVID-19 pandemic, due to which one of the suppliers was unable to deliver the circuit breakers necessary for the panel equipment from Europe on time.

By this time, all pre-design studies have already been completed and design documentation has been prepared, the necessary UPS and most of the materials have been purchased. It was impossible to wait for the situation to return to normal: the renewed server capacities had to be put into operation within a predetermined timeframe under the terms of the contract with the new client of the data center.

Therefore, it was decided to proceed with installation at the site in order to comply with the planned stages of work.

UPS Battery Array in Linxdatacenter Data Center

Beating COVID-19: Planning Flexibility, Teams, and the GUI Factor

First of all, we relied on the well-coordinated work of the team, especially on the “foreman-assemblers” team. I would like to note the contractor’s contribution: Tempesto’s specialists were able to provide the highest flexibility of interaction with us and among themselves in difficult conditions.

For example, when it was necessary to switch power from one source of electricity to another, customers were far from always ready for this procedure, despite notifications. Often they did not allow them to turn off their equipment, because they did not have time to prepare properly – they did not make a backup, did not transfer the systems to a backup site, etc.

Despite this, downtime was almost completely avoided. For such cases, “Plan B” was always ready, with which to keep the installers busy within the current tasks.

In order to complete the project on time, work was carried out daily, without interruptions on weekends. Employees had to work in rather unusual conditions: with constant wearing of PPE and keeping a distance due to the pandemic.

Linxdatacenter Data Center Power Distribution Panel

Competent time management of the team and a clear understanding of the architecture of the future solution helped to avoid significant time losses, and the installation of cable, switchboard equipment and UPS was completed on time.

The status of the official distributor of the contractor for the vendor’s products also helped. When it turned out that one of the UPSs was damaged after unpacking, thanks to direct contact with the supplier, the replacement came in 1 week. If the supplier of the equipment was one, and the installer was another, only on this problem we would lose 2-4 weeks.

results

As a result of the upgrade, which included 4 additional Delta DHP 500kVA series UPSs, power was partially redistributed to new equipment, and the maximum load per UPS unit was ultimately reduced from 49% to 43%.

Overall, the data center’s fault tolerance was already satisfactory, but the upgrade improved it. Previously, if the load of one UPS exceeded 50% of the maximum, then in the event of an emergency, shutdown would be inevitable. For example, when a capacitor assembly crashes in a monoblock UPS, all the systems tied to it fall. In a modular UPS, only one module will fail, and the rest of the elements will continue to work.

Most important: conclusions

The members of the project team have gained important experience in dealing with complex tasks in the face of unforeseen crises.

What conclusions have we come to:

• Supply problems can occur even under normal conditions. In the context of a pandemic and other force majeure circumstances, it is necessary to prepare for the most negative scenarios for the development of events: delivery times may be disrupted more often and for longer periods;
• The work should be carried out according to the schedule-plan, in which any risks are calculated in advance. Special attention should be paid to those nodal points at which the stages intersect;
• In case of violation of the delivery time, it is necessary to do those works that can be carried out regardless of the disruption. If you suspend all work on the project until all the components are received, the work runs the risk of either being delayed or turning into “storming”;
• Modernization of an existing complex is a more complex process in terms of methods than building a system from scratch. Such work is impossible without careful planning and development of contingency plans of action.