Stock up on coffee and headache pills – how to prepare for cyber exercises and win

Stay on coffee and painkillers, globally reconsider tactics in the process and defend your position to the end. The guys from Jet CSIRT tell how their SuperJet team managed to take first place in the international online training to improve global cyber resilience Cyber ​​Polygon 2024.

The guys’ experience will be useful to participants of the main online camp on practical cybersecurity CyberCamp 2024which will take place from October 3 to 5. Representatives of the SuperJet team took part in the development of mission scenarios CyberCamp 2024so catch life hacks first-hand.

All Cyber ​​Polygon 2024 teams acted as a blue team – they investigated the incident using classical computer forensics techniques and a threat hunting approach. The SuperJet team led by Pavel Ivanov, a leading analyst at the Jet CSIRT cyber research group, Jet Infosystems, investigated a targeted attack on an AI organization. In the scenario, MercuryLark released an innovative application using machine learning technologies, but several months after the release, the model’s performance began to deteriorate. It was necessary to find the reason for the deterioration in the quality of the product, study the company’s infrastructure and detect traces of its compromise in order to resume correct operation within 24 hours.

In the end the SuperJet team won 3450 points out of a possible 4020 – this is the best result among 309 teams from more than 65 countries.including the UK, France, USA, Switzerland, Belgium, etc. The teams included representatives of various industries, including the financial sector, e-commerce, education, audit and consulting, medicine and the public sector.

In this article, the guys from Jet CSIRT share tips that will be useful to participants CyberCamp 2024 (have time register!).

“These are our daily tasks.” How to prepare for online cybersecurity training

Pavel Ivanov: “I participated in Cyber ​​Polygon 2021 – then the information security systems used in the infrastructure came to the rescue. This year we had to rely on raw logs, open source and our own knowledge. We decided in advance who would be responsible for what and what tools we would use. The investigation of incidents itself is our daily task, so we relied on our accumulated work experience.”

Alexander Perevalov: “We collected a lot of useful tools, installed an investigation tracker and organized work on one virtual machine.”

“Go in order and look for “flags.” Jet CSIRT – about the tactics of completing tasks

Pavel Ivanov: “We were going to follow the classical method of analyzing artifacts, but quickly drowned in the process.”

Alexander Perevalov: “A thorough analysis of the provided artifacts and entering all the information into the investigation tracker turned out to be a suboptimal solution.”

Artem Semagin: “When we dug into the logs and noted all the suspicious actions, not paying attention to the questions posed, we saw that we were giving our opponents a head start: other teams were moving ahead in points, and we were marking time.

Alexander Perevalov: “After a few hours, we gave up trying to restore the entire painting.”

Valeria Schott: “We realized that we were wasting time. Therefore, we revised our tactics and began to simply rely on questions: we walked in order, looking for “flags.”

Artem Semagin: “Having decided to respond consistently, we began to catch up with our rivals and were eventually able to get around.”

Pavel Ivanov: “It really helped, because the questions themselves seemed to guide us through the investigation and helped us understand the context of the incident.”

Pavel Ivanov, captain of Jet CSIRT, tells how to complete tasks. Beware, spoilers!

We started our investigation with ELK and analysis of user segment events. Having discovered on one of the hosts traces of disabling anti-virus software, uninstalling Sysmon and the telemetry collection agent, we delved deeper into the analysis of the host’s disk image. For this we used a prepared Toolset, including FTK Imager and Arsenal Image Mounter. To speed up the analysis, we collected a triage from the mounted disk using KAPE and began to study the resulting artifacts. Eric Zimmerman's tools (EZ Tools) were chosen to parse forensic artifacts.

First of all, when analyzing triage, we studied Evidence of Execution artifacts. We were able to find traces of the attack in the AmCache, AppCompatCache, and Prefetch files. Of course, we could not do without analyzing the registry, the MFT table, UsnJournal and other artifacts. And for a convenient and quick analysis of events, we used the hayabusa utility, which also highlighted several traces of an attack using the built-in set of Sigma rules.

As a result, it was possible to establish that the initial access was obtained through an interesting RCE – a vulnerability in Telegram Desktop, which we had not previously encountered in our investigations.

The vulnerability is caused by a typo in the list of executable file extensions: for example, instead of pyzw (a file like Python Zip Application) in code with prohibited extensions, pywz was specified. When users received a .pyzw file and clicked on it, the attacker could trick the user into sending a .pyzw file with a spoofed mimetype.

Upon further analysis, it was determined that the attackers used the Mythic C2 framework and dumped credentials. Having gained a foothold on the host via LOLAPPS and passing through a socks proxy, the attackers began an attack on the domain. The compromise occurred through AD CS – as a result of the attack, an account from the gitlab service was compromised.

When analyzing gitlab events in ELK, we found changes to the CI/CD pipeline through which the attackers developed the attack. To understand the attackers’ actions, the repository itself was provided for analysis, in the history of whose commits the further development of the attack lay hidden – the attackers switched to the secret storage and the image registry.

Next, we analyzed the image of the compromised container – analysis of the container image can be carried out using the container-diff utility or even manually, using a notepad, viewing the changes in each layer.

Afterwards we investigated the tetragon events by looking at the ELK logs from the k8s source. Both the events of creating pods and the events of the containers themselves helped us. At this stage of the attack’s development, the attackers chose interesting techniques: reconnaissance using kubectl and creating a privileged container with further escape to the node itself. And the tools used were kdigger, kube-hunter, kubescape, peirates.

The final chord was the compromise of the airflow data orchestration system and the S3 bucket. Here the analysis was again performed using ELK and corresponding events from the same sources.

Dispute with support about commas. What do you remember most?

Pavel Ivanov: “Where would we be without arguing with technical support about “stolen” points?” Cyber ​​Polygon 2024 had a very strict response validator that did not forgive even the slightest deviation. So, a couple of inaccurate answers given late at night, with a colon instead of a semicolon (at night they are really indistinguishable), at first very upset us, and then, on the contrary, encouraged and motivated us.”

Artem Semagin: “Yes, everyone remembers the moment when we were just climbing the standings and made an offensive mistake – we indicated IP:port instead of IP;port. But typos when passing “flags” are inevitable when you analyze logs and artifacts in the middle of the night! We tried to appeal, but to no avail.”

Alexander Perevalov: “Towards the end we even stopped writing disputes, because the support completely ignored us. But I'll tell you about something else. It was quite unexpected to encounter tasks on k8s, which we rarely encounter in practice, so we had to super quickly read some articles and actively google to understand what they were talking about. I also remember a task where I had to extract a piece of binary code from a powershell script, when any attempt to open the script for reading caused everything to freeze and nothing worked. But we did it!”

Live call for a day and resources for BM. Lifehacks of the winning team members

Alexander Perevalov: “It is important to follow the path laid down by the developers. Most likely, it will be shorter than the one you came up with for yourself. If we talk about investigations in general, then write down the most detailed chronology of events, as well as the place and method where and how you found this or that artifact – you will probably have to return to it.”

Pavel Ivanov: “The main advice is to check whether you use a colon or semicolon. But seriously, in such large tasks with a different set of artifacts, it is important not to lose the overall picture of the incident, generate and search for hypotheses, and be able to quickly search for information.”

Artem Semagin: “I advise you to immediately establish communication. A 24 hour live call is a challenge. Of course, everyone could disconnect and connect periodically, but someone always had to remain active in the investigation. It is also important to stock up on hard disk space and resources for the VM. To solve the problems, the organizers prepared a VM image with ELK, but it was almost impossible to work in that VM with default settings.”

“There’s no coffee left, no headache pills either.” How Cyber ​​Polygon 2024 really went

Artem Semagin: “It was cool and interesting, but hard physically and mentally.”

Pavel Ivanov: “There was no coffee left, no headache pills either, which means that the event was as eventful as possible! I hope that next year the organizers will spoil us with the same powerful script and interesting questions.”

Roman Drankov: “I liked the task with a lot of infrastructure and various logs for analysis – I would like to dig deeper into it, but I didn’t have enough time. It’s also a shame that technical support let us down – we need to be more responsive!”

Alexander Perevalov: “The support was really upset – it’s unclear why they sat online all day if they obviously weren’t ready for disputes. The mission scenario leveled out the negativity – it really was cool. Thanks to the organizers – we’ll be back in a year!”

The full composition of the SuperJet team: Pavel Ivanov, Artem Semagin, Valeria Shott, Alexander Perevalov, Maxim Kishmereshkin, Roman Drankov, Pavel Davydov, Daniil Kiryakov.