how we turned the security analyst’s work system upside down and what came out of it

How does a standard response typically occur in a SOC? The analyst received a notification about an incident from some security system, looked at the SIEM logs to collect additional data, notified the customer and compiled a short report for him about what happened.

What's next? And then he switches to a new incident or waits for its occurrence. In extreme cases, participation in incident investigations and development of new attack detection scenarios.

This is the “textbook”, standard work scheme for an analyst in a commercial SOC. This is neither good nor bad – it is, as a rule, just a given. However, the format of a security analyst’s actions can be made radically different, and thereby help an ambitious specialist grow faster and wider, and the customer company further strengthen its security.

I, Nikita Chistyakov, team leader of the security analyst team at Kaspersky Lab. Below the cut, I will tell you how and why at Kaspersky I help create new working conditions for security analysts and how these conditions help them become a kind of Vitruvian security specialists of the Renaissance.

If you are an analyst yourself or generally work in information security, you want to find out what hidden potential the profession has and how to unlock it, this article is for you.

Why did we need to develop security analysts

I joined Kaspersky Lab about 12 years ago and have been constantly growing with the company throughout the decade. Each stage of this joint growth is another domino in a series that led to the need for the company to develop mature security analysts.

year 2014. The information security team lacks a product to protect against targeted attacks, and we, together with our R&D, are launching its development. One such product is our Kaspersky Anti-Target Attack (KATA) platform. Soon, in the absence of a SOC, we are our own monitoring analysts and are actively analyzing the first alerts from KATA and other sources. There are few of us, there are more dens, and there will be even more; This means that more mature processes are needed.

2015 It became obvious to us that it was time to create our own SOC, which would work both for us, as an internal customer, and for external clients. Therefore, we participate in the formation of the SOC team within R&D – we help recruit the company’s first SOC analysts. Telemetry coverage is growing, *DR services are developing, and threat monitoring is now ongoing 24/7.

2020 The SOC has grown to over fifty people, its analysts are engaged in monitoring and primary response to incidents, and complex internal cases are transferred for response to specialists, managers and architects of the information security department. Information security employees are masters of their specialties, but responding to incidents for them is often a non-core task; they have to literally take time off from their own important work, which is why information security projects on development, auditing, and implementation of requirements can be pushed back and stand idle.

At the same time, the amount of telemetry is still growing, searching for information about attacked resources, servers or accounts can take several precious hours, and the response in the worst case can last a month. Therefore, a new level of maturity was required – a dedicated group of experts, each of whom would be a full-fledged incident handler and would be able to build effective response processes. So that, upon receiving information from the SOC about any complex incident, such a specialist can quickly analyze it, determine the criticality of the situation, and then either work on it independently, or assemble a virtual response team from different experts in the company, distribute tasks among its participants, to reduce potential damage. And based on the results of solving these problems, such a specialist, together with his colleagues, would develop an optimal plan for mitigating similar incidents in the future and help implement it.

Here is the last domino in our row – a security analyst reborn in a new role. Or rather, the whole team.

Why does the analyst himself need such a transformation?

Let's go back to the “textbook” analyst we talked about at the very beginning.

Such a specialist usually has a limited SLA to process the incident and explain in the report how to mitigate it. And he, being in the same “textbook” pipeline, is not involved in helping to create and improve products, cannot help modernize the infrastructure and architecture of information security, does not set development tasks for R&D, and generally cooperates little with colleagues from other departments.

I repeat, this is neither good nor bad: but the reality is that after drawing up the report, the analyst has no leverage whatsoever over the customer’s incident mitigation processes. And if you dig even deeper, such an analyst may have professional dissonance: even if he has a great idea on how to mitigate the consequences of an incident and prevent similar situations in the future, he is unlikely to be able to implement it.

Therefore, the tasks of such a specialist often come down to rather monotonous work and monitoring telemetry 24/7, writing correlation rules. At best, providing minimal recommendations.

It turns out that an ambitious specialist may be exhausted by monotonous work, tired from lack of sleep and not grow in the profession as quickly and diversified as he desires. What are the reasons for transformation?

How we contributed to the transformation

Naturally, first you need a technical base – and we have everything you need: from classic hardware servers, information security systems and containerization orchestration tools. But hardware alone is not enough if the processes are not in place: we worked on them with the Threat Intelligence and Threat Hunting teams, expanded ways to automate the response and other aspects of the analyst’s work, and determined how audits of SOC processes will take place.

The next logical step was to form the very teams that would participate in responding to complex incidents – with experts of the highest level, including administrators of IT and information security systems, SOC, members of our global analytics teams (GReAT), AMR, CFR, Forensics. In general, for an incident handler analyst, leading such a team is a serious responsibility that requires experience and expertise.

As the last important step, we gave analysts the opportunity to participate in the restructuring and refinement of existing IT/IS processes, services and products, and also included them in the development of new ones. After all, the information security department is one of the key customers for Kaspersky Lab’s R&D, and our IRT analysts have become real product visionaries.

Less global, but humanly pleasant: we replaced 24/7 shifts with a regular 8/5 schedule in the Moscow office. Of course, it was not possible to completely eliminate night alarms; this is simply impossible for obvious reasons. The possibility of getting up at night due to a sudden incident remains, but at least you no longer have to sit on shift at night waiting (often in vain) for this incident.

This is not the only comfortable bonus: the company helps analysts (and employees in general) relocate to Moscow. Compensates for relocation, accommodation, realtor, and also issues lifting allowances.

What Transformed Analysts Do—And How Fresh Challenges Help Them Grow

So, what are the tasks facing our security analysts?

IoC detection

The basis of the basics, the work of an analyst is always somehow connected with identifying IoC. I will tell you further how certain tasks help to identify them.

Response and investigation of complex attacks on Kaspersky Lab infrastructure and services

So, most incidents are investigated independently by an IRT analyst. But in complex cases, the security analyst becomes the head of a virtual team, which brings together key experts – from network technology specialists and SOC representatives to the already mentioned GReAT and forensics.

To effectively respond to an incident, the analyst must be able to communicate with all experts in their language. Therefore, a specialist must understand many areas: the deeper the expertise in information security and IT, the better. Of course, you also need soft skills – and fortunately they develop quickly on these tasks.

Relatively speaking, you need to have a sufficient understanding of network technologies so that, having delved into SIEM, you can analyze specific firewall configurations and detect IoC in network traffic. Then carry out a collection of artifacts and a primary forensic analysis, discuss the lines in the found exploit with a reverse engineer. Next, based on the data received, return the task to the SOC… In general, you understand.

Dealing with incidents after a mitigation report

At the end of the response, the analyst explains to the client in a report how to mitigate risks, eliminate deficiencies in the infrastructure, or, if necessary, rebuild information security processes. Who is this client? The same information security department in which the analyst himself works. That is, having written a report, our analyst does not finish working with the incident, but only begins it: he will have to directly participate in mitigation. Depending on the situation, he may help colleagues or even be the main person responsible for mitigation.

Thanks to this work, the analysts themselves are aimed at writing clear and detailed reports on mitigation: the entire information security management will then work on these reports, including with the involvement of the analyst himself. And diverse tasks help the analyst to develop deeply not only in monitoring, but also in building architectures, in project management, in DevSecOps…

Organization of automated response processes, development of IR playbooks

Automation is one of the central tasks for a security analyst. The main indicator of response effectiveness is a reduction in response time and, accordingly, a reduction in potential damage.

To automate playbooks, we use scripts and portals for response. So, in addition to classic automated playbooks for email threats, malware or network attacks, we automate integrations between internal services: for example, MDR as a telemetry source, TR database, KUMA and CMDB and other systems. All four systems are linked within the same playbook, and if we detect malicious packages on developer machines or assembly pipelines, then when the signature is triggered, KUMA automatically signals the CMDB via the API to send a letter to the affected employee or, if possible, block the pipeline and notified the service manager. Plus, we have an upcoming release of our own SOAR platform (Security Orchestration, Automation and Response): we are reaching a new level of maturity in the field of automation.

It is also worth noting the group’s analysts’ own developments. To optimize resources and response time within the group, we are developing frameworks that allow us to reduce time costs (in some cases by a multiple) for many routine processes that cannot be avoided during response to most incidents. Many of the group’s internal developments, after entering production, begin to be in demand among colleagues in other related departments.

Threat Intelligence and Threat Hunting

Threat Intelligence literally means: collection and processing of information about current threats on the market, analysis of attacks and vulnerabilities.

We are constantly in touch with our colleagues from other departments and other companies, we ourselves write parsers of interesting sources of information about Internet and DarkNet threats – and, for example, when we receive information about some fresh vulnerability from colleagues from Threat Research, the analysts are right there according to the procedure, they begin to analyze it. Conduct an applicability analysis. They look for traces of compromise using known IoCs. We are figuring out what will happen if someone with an exploit for this vulnerability tries to attack us, and how to avoid or detect this. Develop rules by which monitoring teams will identify potential attacks.

In general, everything is structured so that information arrives as quickly as possible and is processed as automatically as possible.

Threat Hunting is a more proactive thing. Haven't we already been hacked in some advanced way? Maybe we just didn’t notice, and the attacker is already inside our system? This is a paradigm of healthy paranoia: we need to test various hypotheses, look for ioki, actors in logs, traces of attacks. Moreover, we really do have attackers inside: our own Red Team constantly tests our strength.

Naturally, TI and TH work in conjunction; one practice is impossible without the other. These concepts are not new, but they greatly change and develop the analyst’s expertise: he is now proactively looking for threats, and not just reacting to incidents that have already occurred.

Development and implementation of scenarios for detecting complex attacks in SIEM

Security analysts at Kaspersky write correlation rules for our own SIEM, which is called KUMA (Kaspersky Unified Monitoring and Analysis Platform). To find a threat, it is obviously useful to first look for suspicious events.

SIEM systems do not stand still, and the more they develop, the more complex correlation rules an analyst can create for them. We have already begun to use machine learning technologies to identify specific attacks that are not always detected by classical methods: our SIEM already allows us to describe complex correlation rules that detect an attack through three, four, five dependencies.

Already now, correlation rules need to be directly linked to response rules. This is critical for security: I remind you that the best response is a quick response.

Purple Team Process Development

I have already said that we have our own Red Team: we constantly interact with them as part of a full-fledged Purple Team. Sometimes the guys from the red team attack us according to the classic scheme: an unannounced attack from the Red Team occurs, we defend ourselves, and after it both sides write reports. The “Reds” tell us how they broke us, and we tell you which attack indicators we noticed and which we didn’t notice, and based on this we draw conclusions.

We also often create a joint virtual group to emulate a threat. If we believe that there is a method to exploit a vulnerability or develop an attack that is not covered by our monitoring, then we call the red team. They emulate the attack, we watch the telemetry, find out what we are missing, change the processes, the guys repeat the hack, and so on – until we understand that everything is covered by telemetry and detection rules.

The red team is also leveling up as part of the same activity: looking for new paths and loopholes. For example, we are testing tunnel detection. The guys write to us in the chat: they built a tunnel with such and such parameters, do you see? We see. They leave and come back later: okay, they built it with different parameters, see? It turns out that now we can’t see it, which means it needs to be corrected.

Development of cyber training grounds for testing protection against attacks

The cyber range is an environment for training and training analysts, and it is not just two servers deployed on a hypervisor, as is often the case. We have a more advanced approach – we take the Infrastructure as a code approach and use Terraform, Ansible, Vagrant to add as much automation as possible, making the environment highly customizable and efficient.

In 2024, we plan to make sure that any infrastructure – at least three, or at least thirty servers with three domain forests and in various configurations – can be deployed not the old fashioned way, but only through script developments and config files.

But most importantly, at the cyber testing ground it is convenient to independently, without the participation of the Red Team, simulate attacks on a copy of the real infrastructure through automated frameworks. Or you can add to such a copy applications, servers, services that do not yet exist in the real infrastructure – and then training becomes as proactive as possible: here you can already run vectors and look at triggers, which makes it possible to write rules for the future and stay ahead of future threats by developing Threat Hunting.

Cooperation with R&D

A security analyst at Kaspersky fights the most advanced threats. This means he needs the most modern means of protection. And so he becomes a security evangelist.

Whenever possible, we in information security try to use only our own developments. Therefore, analysts work with R&D to complement existing products and develop new ones. This is not a one-time situation: an idea is born in the security department, and R&D brings it to life. For example, this is how the above-mentioned KUMA appeared. One of our analysts, by the way, headed its development department. It's the same story with our EDR, Endpoint Detection and Response.

Another striking example is our KATA. The same one whose appearance my comrades and I were so happy about in 2014. Back then we had few capabilities to detect network and email threats, so we put forward many demands on the nascent KATA and were constantly in contact with R&D. And soon after its release, we caught powerful targeted attacks with its help. The loudest of them is probably Duqu 2.0.

From KATA we needed a global IDS for the entire infrastructure, mail analysis, sandbox, working with YARA rules, processing telemetry from endpoints. Soon, our analysts needed tools for remote response, artifact collection, and forensics. Now almost all KATA functionality has been implemented and is being developed in accordance with our requirements.

We first test all the services and products that we develop on ourselves, and when they reach maturity, we bring them to the market. So being evangelists in this regard is also important – you partly determine the development of future client products.

Conclusion

In general, as you can see, there are many tasks, they are complex, varied and interesting. But in this context, I like the proverb that you should ask not for a small burden, but for strong shoulders. An ambitious expert who already has three to five years of experience in information security or SOC, or a seasoned Threat Hunter with knowledge of Python or Go and a deep understanding of Linux, becoming a security analyst, will be able to develop these skills to the highest level. I’m not afraid to seem immodest: my colleagues’ level of expertise is comparable to the Head of SOC of a number of other companies.

We are actively looking for experts to join our team and are ready to invest in their development. Even if certain skills are lacking, the company will help improve them: in my team, analysts, in parallel with their work, study in various courses, including from the same SANS Institute.

So, if you are interested in my story and want to try yourself in the role of a Renaissance security analyst, respond!