CrowdStrike update crash or blue screen day
The dry facts are presented in publications CrowdStrike itself. On Friday, July 19, at 4:09 a.m. UTC (7:09 a.m. Friday Moscow time and 11:09 p.m. Thursday in Austin, Texas, where CrowdStrike is headquartered), an update to the client software (the so-called sensor) for CrowdStrike's Falcon product was released. Several sources say that the bug was present in the kernel driver that powers the Falcon sensor. The company denies this, calling the failed update a “configuration update.” These are essentially instructions for the security software that are sent to clients regularly – several times a day. The update files, with the now well-known name “C-00000291-*,” contained instructions for data execution control using the standard named pipes mechanism, or named pipesThis mechanism can be used by malicious software, and it is logical that security software contains methods for detecting unwanted activity.
The update contained a certain “logical error” that caused the OS to crash. After 1 hour and 18 minutes, the error was noticed by the developers and the faulty update was replaced. Accordingly, the computers that had managed to receive the update with the error failed: due to the inevitable delays in this process, a situation arose when corporate systems (for example, cash registers in supermarkets) still partially worked. In the case of physical PCs, it was sometimes possible to return the system to operation by a simple reboot. The computer connected to the Internet, downloaded the corrected update instead of the erroneous one and restored operation. If it managed to do this before the next crash to the blue screen, which is why the company recommended use a wired connection to the Internet. But with high probability, administrators was required boot into safe mode, go to folder C:\Windows\System32\drivers\CrowdStrike and manually delete files starting with C-00000291-.
For large companies with a large fleet of systems, even a forced reboot of each computer or server is a bad scenario, especially manipulations in safe mode, manually or using a later provided USB drive or special utilities from Microsoft. The task can be greatly complicated by using encryption with the standard BitLocker tool – in this case, you need to get the so-called recovery key for each system. If the server distributing BitLocker keys itself suffered from a failure, the problem becomes even more “interesting”. Virtual machines running Windows were also affected – to bring them to their senses, a reboot through the control panel is usually enough, but sometimes, according to eyewitnesses, this had to be done 15 times in a row.
Let's get back to the discussions on the Internet. It is worth immediately excluding talk about the advantages of Mac OS and Linux. Yes, only Windows systems suffered, but only in this particular incident. In April, another software update for the same CrowdStrike brought to the crash of Linux hosts at least at one client. It is impossible to speak about the uniqueness of such an incident. In 2010, McAfee released update that disabled computers running Windows XP SP3. In 2012, Avast's solution deleted on client computers, the critical system file tcpip.sys. The CrowdStrike incident thus stands out only due to the scale of its consequences. Regularly delivering up-to-date threat information to clients with appropriate protection methods is a complex process in which the possibility of error cannot be ruled out. The task of security software developers is to find organizational and technical solutions that ensure that such errors do not affect the functionality of client infrastructure.
Another narrative in the wake of the crash is about the insecurity of C/C++ development. But it is too early to talk about this, since the exact cause of the crash is unknown. Attempts at technical analysis have been made, but they are slightly contradictory. This one thread on the social network X/Twitter has gained 28 million views, but its legitimacy disputes renowned security researcher Tavis Ormandy. No less renowned researcher Patrick Wardle offers their version of events. All of them are based on the analysis of the Windows system logs after the crash, the probable cause of which was an access to a “forbidden” memory area. But what caused such an access is unknown at the time of writing. The version that an empty file filled with zeros was accidentally sent to clients was also not confirmed. To find the cause, it is necessary to analyze in detail the logic of the proprietary software, which, most likely, none of the independent experts will do. It remains to be hoped that CrowdStrike will keep its promise and will report the results of the investigation in as much detail as possible – this will help the industry as a whole to ensure that such incidents do not happen again in the future.
Another expected consequence of this incident was spreading malware under the guise of “solving the problem with CrowdStrike.” Not surprisingly, in the very first message about the incident, CrowdStrike’s CEO urged customers to communicate only with representatives of the organization.
What else happened?
A recent study by Cloudflare reports that the average share of junk and malicious traffic has reached 6.8%, up from 6% a year earlier. Malicious attacks account for 54% of this traffic, DDoS accounts for 37%, and the rest is requests from blacklisted IP addresses.
On Thursday, July 19th also happened Unrelated to CrowdStrike, an outage affecting Microsoft 365 services.
In public domain got there 15 million email users of the Trello service.
