how we saved our information security services after the departure of foreign vendors

The battle is over, the fire has withered and there is nothing left,

And we live, and we are lucky to spite you.

Agatha Christie, “Like at War”

m/f "Madagascar"
m/f “Madagascar”

For many Russian companies, the departure of foreign software and hardware vendors was, to put it mildly, a blow in the gut. Service providers had a doubly hard time: they had to provide oxygen mask necessary software not only for themselves, but also for their customers. A large part of our information security services was based on Western solutions. Something began to work with a truncated set of functions, and something simply turned into a brick due to the revocation of licenses. There was no time for a long search for an alternative, some companies were already under attack. What we encountered when foreign vendors began to revoke licenses and curtail technical support, how they were looking for a replacement for them and what they did when they were left without an orchestrator – we will tell in this post.

It hurts but bearable

Almost everyone who is now reading this post has encountered some of the problems described. The first and most obvious one was related to the closure of local offices of foreign vendors and the termination of full-fledged technical support. Therefore, we decided to start with people: we strengthened our technical teams for development and operation. So we got rid of the urgent need for third-party expertise and now we can complete these tasks on our own.

With hardware platforms it was more difficult. Here we were helped by an architectural approach with multiple redundancy of each of the components (in “ordinary life” this allows you to distribute the load on the platform and make the services more stable). In addition, we designed the services from the outset with a solid supply of hardware resources to ensure their high performance. Thanks to this, the platforms we use can “live autonomously” for three years.

With software licenses, too, of course, there were difficulties. As in the case of hardware capacities, we purchased licenses with a large margin and plans for the future. Some solutions within the platforms are based on Right-to-Use licensing, that is, there is simply no technical possibility of revocation. So for now we hold on and look for alternative software.

But you can upgrade … Well, how can you? Even if you managed to download the update in a roundabout way, there is probably no trust in new versions. Under the current conditions, the emergence of undeclared opportunities in them seems very likely. Therefore, now it is worth thinking 100 times before downloading this or that update. Of course, if you just need to install an update, we conduct functional and load tests to minimize the risks, but, alas, it is impossible to completely eliminate them.

Then there was the question of iron. How to purchase replacement equipment (spare parts) and to expand computing and network capacities? There are few options here: find alternative approaches to building supply chains or work out parity replacements for server and network equipment. So far, the existing margin of safety saves: a large resource of equipment will allow replacing each of the components for about three years.

How exactly was it

Now let’s look at specific examples. Two of our services were based on American Fortinet solutions: FortiMail was used for email protection (SEG), and FortiGate was used for network threat protection (UTM). The company promptly revoked its licenses at the beginning of March, although until recently it claimed that it had no plans to leave the Russian market.

UTM turned into a brick

The network threat protection service instantly turned into a pumpkin due to licensing features (one of the few, it was not implemented according to the Right-to-Use principle). We worked with the vendor according to the scheme provided for MSS providers – cloud licensing. Points were tied to this license for use by our customers’ FortiGate virtual machines. As a license server, a locally deployed FortiManager in HA configuration was used, which constantly communicated with the FortiGuard cloud to validate the license and write off the used balance of points. This made it possible to more flexibly configure the service for each client and pay only for the set of functions that is in demand. Additional options, if necessary, could be connected and disconnected online. In addition, this type of license allowed us not to spend extra resources – neither ours nor the client’s. In general, all this was very convenient for the time being, for the time being. If it was impossible to check the validity of the license, the firewall stopped routing traffic altogether, making customer information systems inaccessible.

Fortunately, Fortinet, although it was the most demanded, was not our only vendor for the UTM service. In particular, we were at the final stage of testing the domestic UserGate. As a result, we implemented an emergency switch to a Russian vendor within one or two days. We were helped by the fact that we already had parity solutions for replacement. We also had (and still have) Israeli Check Point in our arsenal.

Almost no SEG

The email protection service was a little more fortunate – the letters continued to go through. But due to the revocation of licenses, the solution lost the ability to update its signatures and rules online. This meant that we could not guarantee high-quality e-mail protection to our customers. Yes, we had the technical ability to update some of the functional protection modules offline, which would completely devalue the benefits of a cloud service.

But still, SEG did not turn into a brick, giving us the opportunity to work out a replacement for the basic solution. We gradually dealt with this issue, but there was no such ready-made answer as with UTM. At the moment, connection options from Russian vendors are being worked out.

VM left without a cloud

Another service that turned out to be unavailable after the revocation of licenses is the Vulnerability Management (VM) service. For it, we used the American Qualys cloud scanner. Everything was relatively simple here: no license, no access to the scanner. It was obvious that in the new conditions it was possible to replace the solution only with a domestic cloud. We are already working with one of the Russian vendors, so customers are not left without perimeter scanning. So far, the service is focused more on mass medium and large businesses. However, in the near future there are plans to expand the cloud to meet the needs of a large Enterprise segment.

How we were left without an orchestrator

So, we were ready to migrate clients, but at that moment we caught an accident with our high-level orchestrator, through which we deploy services for clients and change service parameters, including replacing one solution with another. Probably, we would not have avoided an accident in such conditions, no matter how hard we tried.

First, we were hit by a sharp influx of clients who urgently needed protection from everything at once. In addition, to connect them, we had to allocate additional server capacities, which are usually reserved for the management of cloud components. In addition, we were migrating services to the updated platform, and this required additional capacity. Well, to the heap – new solutions required more power than, for example, the same FortiGate.

In a word, we were left without a tool for managing services within the service platform, there was no opportunity to wait for its restoration: no one canceled obligations to customers. Therefore, the team of our engineers quickly reworked the existing automation descriptors that are pulled from the platform’s external orchestrator into playbooks external to the platform. This allowed us to deploy the service with the same level of automation, but bypassing the orchestrator. In short, we prepared playbooks, agreed on a schedule, and manually carried out work to replace them with alternative solutions.

In general, the deployment of the service is fully automated. At the click of a button, dedicated tenants are deployed at all levels of the platform, a dedicated logical network and computing infrastructure is created, the necessary set of VNF network functions is deployed, interconnections between components within the SDN / SD-WAN parts of the platform are organized in accordance with the design of the desired service, the VNFs themselves are configured, fault-tolerant configurations are created, etc. And that’s thousands of lines of code.

Let’s go without the old rake

The experience of an extreme transition to new vendors was, of course, unforgettable, but I don’t want to repeat it at all. Therefore, it is now obvious that when choosing technologies for our services, we will give preference to domestic solutions.

We also made sure once again that all solutions must be integrated into our own infrastructure and deployed exclusively on our own cloud, so that we always have access to all the technologies that we use. So we will always have full control over the infrastructure of services. However, as we wrote above, there is always a risk of various NDV – this risk is reduced by FSTEC certification.

And, of course, we must pay tribute to our clients, who, understanding the situation, also met us halfway, did not swear much and gave the necessary feedback. Now the problem of availability of services and security of clients is solved. Imports have been replaced %@&#as the saying goes!

Similar Posts

Leave a Reply