The pain of a manager when implementing security systems in a company or how I implemented MFA

Not long ago I was working on a project to implement a system M.F.A. in an IT company. Now I am watching how other IT companies are implementing MFA systems and I see many mistakes by managers, which subsequently affect the negative attitude of employees towards the system as a whole and towards the departments that implemented it in particular. I want to share the pain that a manager faces when implementing such systems in a company with 2000+ employees, and also talk about the methods that allowed them to achieve their goals.

I don’t see any point in talking about the concept of MFA or its subtype (2FA), because I saw a million articles on this topic on Habré. So I'll get straight to the point.

After modeling threats in the company where I once worked, I decided to implement various information security systems in order to neutralize these threats. One of these information security systems was the MFA system.

I started studying the market, looking at what Russian vendors were offering – we are “for import substitution”! On the market I discovered 2 types of architectures for the functioning of MFA systems: self-hosted and service. Let's start with the last one.

The MFA service architecture assumes interaction with the authentication server on the service provider side. The architecture itself looks something like this (let’s look at it quite superficially using the example of access to a VPN server, without going into too much technical detail):

1. The user uses the VPN client to connect to the VPN server.

2. Enter the username and password for the VPN account.

3. The VPN server sends a request to the Radius Server.

4. Radius Server accesses an account provider service, such as Active Directory. This service checks the validity of the user's entered data. If everything is OK, then a message about the validity of the entered data is sent to the Radius server. At this point, the first factor of authentication ends.

5. Radius Server sends a request to confirm the second factor of authentication to the authentication server located in the service provider's data center.

6. The authentication server waits for the user to enter the OTP (code generated by the application, SMS code, etc.). The user presents the OTP to the server (code generated by the application, SMS code, etc.).

7. When receiving a valid OTP from the user, the authentication server sends a message to the Radius server about its validity.

8. Radius Server allows the VPN server to grant access to the user.

What conclusion can be drawn about the use of such an architecture? Convenient, fast, little hassle for the company’s IT and information security employees when implementing it, the cost of acquisition and maintenance “on the spot” is much less than the implementation of self-hosted solutions. And yes, they also thought about data security – logins and passwords are not transferred to the solution provider. In general, everything would be fine, but I saw one big problem in this decision.”BUT” The use of such an architecture creates additional threats to my company due to the fact that we must give a third party direct access to the internal components of our infrastructure located in the protected network loop. And not just to whom, but to the Radius server itself, which distributes all permissions to internal services. Then all the possible attack scenarios that could be carried out immediately came to mind. For example, a hack of our supplier will certainly lead to a hack of our entire company (there is a lot of talk about this now on various sites and forums – called supply chain attacks). Such a scenario carries quite large risks that I, as the person responsible for this, could not take upon myself. Maybe our supplier has everything cool with security and will never be hacked, but I, being a callous and risk-oriented security specialist, until I can check this and verify it personally, will not believe in any pieces of paper, certificates, licenses and statements of the service provider about the security of its infrastructure.

In this regard, I decided for myself that I would consider only self-hosted solutions, since there are also such on our market. I will not give the architecture of self-hosted solutions, because… in meaning it will be the same, only with the difference that all components of this architecture will be under my control, i.e. on my company's network.

So, I started searching for a good self-hosted MFA system solution. The search did not take very long, fortunately there are not many of them on the market. My attention focused on one solution that met all my requirements (self-hosted, the ability to transmit OTP in the form of SMS and codes generated in a special application, the ability for the server to generate one-time QR codes for connecting the user, and a bunch of other requirements about which I will not write in order to reduce the time spent reading the article). I will not tell you the name of the MFA system I chose, because… I'm not advertising anything, I'm just sharing my experience.

Next was the piloting process. The pilot took place in 3 stages with a constant increase in the load on the server and testing of various system functions with different loads. At the first stage, we piloted the system with a test group of 20 users, at the second stage with a group of 50 users, and at the third stage with a group of 100 users. At the same time, at each stage the load on the system per unit of time was checked, i.e. all users simultaneously requested OTP, or at the same time entered the received codes into the application, etc. In general, testing revealed certain problems, but not critical ones; the vendor managed to fix them in a few days. By the way, we must pay tribute to the vendor – he worked with a bang, did everything quickly, for which I would like to say a big thank you to his employees (if any of them are reading this article now). As a result, the pilot was a little delayed due to problems that arose, but after its completion, I was already confident in the quality and performance of the product and that after implementation in the company everything would work properly and safely.

Next, let's move on to the most interesting part – how I implemented this system and the problems that I had to solve.

Problem one. How to implement the MFA service for a company of 2000+ people, provided that all employees work in different formats (office, hybrid, fully remote)?

The problem was that 2FA first had to be installed on the VPN, i.e. if something goes wrong, the employee will not be able to work in corporate systems (and almost all of them were behind VPN), which will lead to downtime for the organization, financial losses and…

How to solve this problem? The first thought that came to my mind was to carry out the implementation not just once, but gradually, in groups, in order to localize possible problems. The first thing I did for this was to take the employee directory, look at which groups in LDAP their departments belonged to, and began to cut into groups of people + – 50 each. the idea was the following: we enter data about specific LDAP groups (which include ± 50 people) into MFA, they receive a one-time QR code by email, which they scan with the application on the device, thereby receiving an OTP generator on the mobile device.

Why was the group of 50 people chosen? It’s simple – potentially, I processed the risks during implementation, and I understood that not all people are friendly with technology and some of them themselves will not figure out what to do, even though I developed for them brief but most accurate instructions (with pictures, arrows , in general, so that even a first-grader can understand) to carry out the necessary actions when connecting. Due to such risks for their processing, I came to DIT and talked with my comrades from technical support. It was jointly decided that if such users appeared, the technical support service would be able to process requests and help set up the system for no more than 50 people per day. When we discussed this, the support guys laughed at me, saying “oh well, the instructions are clear, everyone can connect everything themselves.” But I knew that people are different and did not take risks. Spoiler — when they started implementing MFA, people flocked to support and bombarded them with applications and requests for help with the connection. Of course, not 50 people came a day, but on average there were 10-20 people, and when they reached non-IT departments, then 30-35 people a day. In general, the support staff later thanked me for the fact that the implementation groups consisted of 50 people.

Somehow I got distracted. Let's continue. In connection with this decision, as well as with the architecture that we built for the secure operation of the server (I will not describe it, because it is not correct), in the process we had a bunch of problems with LDAP groups, with groups for access to the VPN itself, as well as with user groups on the MFA service itself. I won’t describe these problems either, because… this is also incorrect. But in general, after much thought and study, all the problems were solved and we began to introduce users to MFA.

We gave users 3 days to connect to MFA, after which we removed them from VPN groups that were allowed access to VPNs without a second factor of authentication. Also, after 3 days the QR code went bad. It would have been possible to leave the QR code for a longer time, but I was afraid of a situation where an attacker would gain access to the mail of a careless employee who did not activate the QR code for various reasons (works from the office and does not need a VPN, is on vacation, works in corporate systems that are behind a VPN, or it simply doesn’t work).

Do you think 3 days is enough for an employee to download the application and scan the QR code? So I think that's enough. Spoiler – 15 percent, i.e. about 300 people DID NOT manage to do this. Support staff and system administrators were forced, in addition to serving users in accordance with the implementation list, to also serve these guys. Support and admins started working until late in the evening.

Problem two. How to explain to users that the OTR delivery method via SMS is less safe and that they need to use OTR generated by a special application, while remaining alive and without a bullet wound in the chest?

First, I will describe the essence of the problem. We all (security professionals) know that the technology for sending OTP via SMS (hereinafter referred to as SMS codes) is more vulnerable than the technology for generating OTP on the user’s device (hereinafter referred to as OTP code). I don’t see the point in telling why it’s not safe, because… There are a lot of articles about this on Habré. If you simply tell people that SMS codes are not safe, but OTP codes are safe and that they should be used, then, at a minimum, you can get a lot of puzzled looks from employees, and at a maximum – a lot of disputes, attempts to prove to you the “unqualified security specialist” that everything is safe, otherwise why then do State Services use them??? In such a situation, you could explain and prove to everyone, but it won’t end well, and you’ll waste 100,500 hours of time. You can prepare a report and present it to the whole company – again, you will waste a lot of time, but you still won’t achieve results, because someone won’t listen, someone won’t come, and they’ll still come at you later with their “truth.” You could make pictures and email them to all employees – many won’t read them, those who read them will still get into an argument with you, and so on ad infinitum. In general, there is no way out of this vicious circle.

The company needed to abandon SMS codes not only to achieve a more serious security indicator, but also to minimize the financial losses that would arise from their use.

Let me explain: the cellular operator charges about three rubles for each SMS delivered to the user. Let's do some simple calculations:

About 70% of the company's employees constantly use a VPN, and only 30% of the employees will not really need it. It turns out that VPNs will be used by:

If you allow the user to choose the OTP delivery method, then about 80% will use SMS codes:

On average, users will request 3 SMS codes per day. It turns out that users will need to send:

This will cost the company:

The company's annual spending on SMS codes will be:

This does not include the fee for the MFA license and its support, the cost of server hardware, etc. This is a lot of money that you don’t want to lose, don’t you agree?

At the same time, I understood that it would not be possible to completely ban SMS codes, because… There will still be people in the company who will come and say:

I use a feature phone and can't download the app

As a matter of principle, I am not going to install an application from some vendor on MY PERSONAL PHONE, you won’t force me

I don't have a phone at all, buy me a smartphone and give me a SIM card so I can use your MFA application

If you think “There are no such people”, “What kind of nonsense?”, then here you go spoiler — I counted about 50 such people during implementation.

I understood that we would either have to buy smartphones from the company for such people (and the cheapest ones cost 5-10 thousand, plus SIM cards for communication – that’s another 500 rubles a month, and we’d have to tinker with it so much that my mother don’t worry), which will lead to even greater losses, or allow them to use SMS codes for authentication (if they have devices) – this option seemed the most logical. If you allow such people to use SMS codes, then another problem arises:

In general, again a vicious circle.

So it was necessary to come up with something that would allow such employees to use SMS codes and so that other employees would not want to use them. It is also advisable to ensure that the company achieves its goals:

• introduce 2FA (so that people do not refuse to use it);

• do not lose extra money (use OTP codes);

• so that no one quits (developers nowadays are very touchy and valuable as a “Faberge egg”);

• so that employees are satisfied (so that they are not negative due to the use of technologies that they did not choose).

To achieve the first goal (implement 2FA), I had to carry out campaigning and propaganda using various corporate services: I wrote letters to the entire company explaining the reasons for introducing MFA and plans for its implementation, promoted the need for implementation at various meetings (meetings, video conferencing, corporate events etc.), spammed the corporate messenger, even created a separate group in the messenger for particularly stubborn employees, where he justified the reasons for introducing such technology. After a certain time, people seemed to resign themselves and were ready for implementation.

In general, even as a callous but still risk-oriented security professional, I understood that the likelihood that someone would deliberately implement attacks to intercept SMS codes in order to penetrate my company was extremely small. And even if they decide to do this, then the offender’s potential in this case should be high, and this is not just any student who decided to play around. There are few such attackers, and their interests are completely different. If Banks and various government services that process much more sensitive data are not afraid to use this technology, then why should I be afraid?

Then I decided to allow users to use any technology, let them choose what is more convenient for them. With this solution I completed the following goals:

As you understand, having allowed people to choose, the majority (80 percent offhand) will choose to receive SMS codes rather than OTP codes. This will entail financial losses, but I had to achieve one more goal: not to lose extra money (use OTP codes).

Then in front of me the question arose: how to allow users to choose any method of receiving OTP, while reducing the number of users (to a minimum) who will receive SMS codes? And all this had to be done in such a way that the user himself was satisfied and continued to cooperate fruitfully with the company. To do this, he himself had to decide on the choice of OTP codes for authentication, despite the fact that he also had another choice. In this case, he remains satisfied, and the company does not lose money.

To solve this non-trivial problem, the following solution came to mind. I have developed 2 instructions. The first instructions were one A4 page, which told which application to download, how to scan the QR and how to use the OTP code. The second instructions for connecting the OTP delivery method via SMS included a mention that this technology is unsafe, and that it would be better to use OTP codes. If at this point the user did not “perceive” this information, then the next instruction was: the user who wanted to use SMS codes for the second authentication factor had to print and sign the application. This statement contained an intimidating phrase a la “I understand that I am choosing an unsafe method for obtaining OTP and I bear full responsibility for any loss of corporate information and other consequences that occur in the event of the implementation of NSD to information processed in the company’s corporate services using my account.” . This should have been the second control point at which the employee should have thought about the fact that he wanted to use unsafe technology and understand the risks associated with it. If here the user wanted to continue, this meant that any beliefs about the unsafety of this technology did not work for him. Then the usual mechanism of the universally unloved bureaucracy was activated, which the average employee had to refuse in favor of a simpler method for him to connect OTP codes. This mechanism assumed the following: the employee had to print out the application, sign it himself, sign it from the manager, sign it from the lawyer, sign it from the head of the information security service and, ultimately, from me. After receiving my final signature, this document was scanned by the information security service and sent to the Service Desk to DIT employees, who switched the authentication method in the application for this user. The result is obvious – 1,120 users wanted to use SMS codes for authentication, as a result, 1,115 users independently decided that they would use OTP codes, just not to plunge into bureaucracy. Of those 50 people who came and said that they could not use OTP codes for one reason or another, only 5 people still went through this procedure and got the opportunity to use SMS codes, the rest still “suddenly found smartphones” and went install the application.

Yes, you are reading this now and thinking “what a bad person, a radish… why be so perverted?”, but try to look at it from the other side. I needed to achieve all my goals. I wanted the users’ experience with the company’s services to be safe, so that they would use MFA without negative emotions, the company would not spend extra money, and the business owners would be satisfied. The work done allowed me to achieve all my goals. I saved about 2,500,000 rubles a year for my company, with which we were able to hire another employee who could make the company’s products and services better. I did not forbid people anything – they acted of their own free will and were guided by their choice. And yes, almost the entire company (99.9%) ends up using a more secure technology than protects important information processed on internal resources. All the best and thank you for reading to the end!