How the Blue Screen of Death Broke Airports and Businesses

On July 19, there was a global outage of Windows PCs and servers around the world. A number of devices displayed a blue screen of death (BSOD) and went into an endless reboot. The cause of the outage was an update to the Falcon Sensor cyber-attack protection system produced by the American information security company CrowdStrike. The problems affected the IT infrastructure of many companies, banks and airports around the world.

CNET Edition indicatesthat the last similar-scale outage occurred in June 2021. Then, disruptions in the infrastructure of the cloud platform of the content delivery network provider Fastly led to problems for many companies and services, including Twitter, Amazon, Reddit, Spotify, eBay, Twitch, Pinterest and others. In the summer of 2021, the problems were observed for 45 minutes.

The faulty software from CrowdStrike affected 8.5 million devices worldwide, or less than 1% of all devices running Windows, reported at Microsoft. The Redmond corporation called the percentage small, but the social and economic consequences broad.

The outage sent Microsoft's stock down 3% and CrowdStrike's down 18%.

CrowdStrike, an $80 billion company based in Austin, Texas, is best known for investigating major cyber incidents, such as the attacks on Sony Pictures and the Democratic National Committee. The firm was founded in 2011.

What happened

The cause of the failure was an update to the CrowdStrike software and an error in its interaction with Microsoft services, including Microsoft 365 and the Microsoft Azure cloud platform. The systems have been partially restored, but some users continue to experience problems.

CrowdStrike CEO George Kurtz was quick to reassure that the outage was not caused by a security issue or a cyberattack. He said the issue had been identified, isolated, and addressed. Kurtz added that the outage was caused by an update for Windows hosts, so macOS and Linux users were not affected. The code flaw was in a single content update for Windows hosts, the CEO explained.

Officials in Britain, Turkey and France also believe the outage was not caused by a cyber attack.

CrowdStrike reported that it had discovered and fixed the issue. The company directed customers to its support portal for the latest updates. To fix the BSOD on Windows 10, you can refer to instructions. CrowdStrike Representatives opened a Reddit thread about blue screens of death after crashes.

Microsoft experts even reported that BSOD can be fixed by rebooting the computer 15 times in a row. This solution is intended for system administrators of virtual machines using Azure and who have encountered a global failure in the operation of PCs and servers on Windows.

Later, Microsoft reported that Microsoft Defender, Microsoft Intune, Microsoft OneNote, OneDrive for Business, SharePoint Online, Microsoft 365, Viva Engage, and Microsoft Purview were back to working correctly. However, there were issues when using PowerBI, Microsoft Fabric, Microsoft Teams, and Microsoft 365 admin center.

Falcon Sensor provides highly accurate detection and automatic protection against potential cybersecurity threats in real time. The problem was in the code of the workstation software. An update to the files “C-00000291*.sys” in the folder “C:\Windows\System32\drivers\CrowdStrike” led to BSOD of hundreds of thousands of PCs worldwide. The root cause of the global failure has been eliminated.

Current solution from the manufacturer:

Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

The latest CrowdStrike troubleshooting tips and information recommends get in your blog or at support portal.

A CrowdStrike software update broke server builds on Debian and Rocky Linux in April and May of this year, but only a few customers noticed. Security developers dragged their feet on closing tickets for weeks and issued formal replies to customers, taking a long time to release the necessary patch. Even then, users noted the unsatisfactory level of CrowdStrike testing and technical support.

The developer's security services are used by 29,000 companies worldwide, including 500 from the Fortune 1000 list. It became known that the majority of the company's clients prefer Windows, and CrowdStrike technical support responds quickly only in the event of a global problem.

Consequences

The CrowdStrike outage has affected systems in many countries, including the US, UK, EU countries, India, New Zealand and Australia. This includes flight and hospital cancellations, payroll freezes, TV channels being shut down and more.

Numerous airlines suspended flights on July 19, including three major U.S. carriers: United Airlines, Delta Air Lines, and American Airlines. United said it would detain planes at airports until systems were restored. Other U.S. airlines — Frontier, Allegiant, and Spirit — also suspended flights due to the Microsoft system failure.

There were also problems in Europe, with Irish airline Ryanair advising passengers to arrive early for boarding due to a “network-wide disruption.” The decision to suspend operations accepted and at the Dutch KLM, a system shutdown made it impossible to service flights.

British Airways, Wizz Air, Turkish Airlines, Eurowings, Lufthansa and Qantas also reported problems. According to the analytics company Cirium, on July 19, was cancelled about 7 thousand flights – 6.2% of all planned worldwide.

Representatives from Belfast and Singapore airports toldthat due to the BSOD on the board, employees used whiteboards and marker writing to inform about flights. Similar measures came running in India. In some countries, airline tickets were filled out by hand. The systems were later restored. Guardian writesthat the problems affected airport systems from Amsterdam to Zurich and from Singapore to Hong Kong.

The fourth largest US airline, Southwest Airlines, has crashed did not affectbecause the carrier uses Windows 3.1 on its systems. The operating system, released in 1992, does not receive updates. Southwest also uses Windows 95 for its workforce planning system.

Alaska State Police reportedthat a global outage has affected the state's emergency communications line.

In Great Britain suffered health systems that doctors use to make appointments, view patient records, order prescriptions, and issue referrals. At least two German hospitals cancelled elective surgeries. Israel's Health Ministry announced that the global outage affected 16 hospitals. The incident also influenced for the operation of banks, the postal service and emergency response systems.

Unavailable turned out to be information about the arrival of New York City subway trains. However, the disruption did not affect the movement of trains, but it did make it difficult for subway workers to see the location of trains. British railway company Govia Thameslink Railway (operates Southern, Thameslink, Gatwick Express and Great Northern) warned passengers about possible delays in train arrivals.

British TV channel Sky News interrupted broadcast after a Windows crash. The channel switched to showing archived footage and briefly showed an error message. Problems also occurred on India's NDTV, Australia's Channel 10, SBS, ABC and other channels.

The London Stock Exchange website experienced problems, making it impossible to publish news. Other exchange services continued to operate normally.

In Japan suspended work about a third of McDonald's fast food restaurants. The problem was related to problems with the chain's cash registers. The Australian grocery chain Woolworths was also affected by the glitch.

US delivery companies FedEx and UPS have warned of delays in fulfilling orders due to a glitch. Problems have appeared in the systems of several US government agencies.

Tesla CEO Elon Musk said the outage had paralyzed the car supply chain. He reportedthat the auto company stopped using CrowdStrike software and removed it from all of its systems.

British users reported problems with Visa and BT services, as well as supermarkets, banks and online gaming platforms. Metro Bank pointed to connectivity issues in the UK, while Santander reported problems with card payments.

Problems with banks and payment systems have arisen in Australia and New Zealand. Clients of National Australia Bank, Bendigo Bank, Commonwealth Bank of Australia, Bank of New Zealand, ASB Bank and other banks have experienced disruptions in banking and electronic payments.

Cybersecurity expert and Microsoft Australia Regional Director Troy Hunt named The scale of the latest outage is unprecedented. According to him, the incident can be called the largest IT outage in history.

British Computer Society (BCS) experts predicted that systems could take days or even weeks to restore. The occurrence of BSODs and endless boot-ups suggests that recovery could be difficult, said BCS fellow Adam Leon Smith.

Stephen Murdoch, professor of security engineering at University College London, believes that many companies will find it difficult to fix the problem quickly. The failure occurs before the computer connects to the internet, so there is no way to fix it remotely, he says. Murdoch adds that companies that have cut their IT staff or outsourced their IT tasks will face difficulties in solving the problem.

Australian Home Affairs Minister Claire O'Neill warnedthat attackers are trying to take advantage of the global CrowdStrike outage to rip off small businesses by offering them fake fixes. She reported attempts to conduct phishing attacks on firms.

A faulty CrowdStrike software update continues to impact businesses and individuals around the world despite promises from CrowdStrike and Microsoft to fix the issue, reported Neowin. Microsoft CEO Satya Nadella wrotethat the company is working closely with CrowdStrike and the wider industry to resolve the issue. The incident is still affecting the UK's National Health Service.

Impact of the failure on Russia

The Russian Ministry of Digital Development reported that the situation with Microsoft highlights the importance of import substitution of foreign software. The department stated that they had not received any notifications about failures in Russian airport systems. Sheremetyevo Airport explained the stable operation of the infrastructure by the implementation of its own developments and solutions.

The global failure did not affect Russian nuclear power plants, since Rosenergoatom operates on import-independent software. VTB and Sber systems are functioning normally.

There are virtually no CrowdStrike solutions in Russia, so the global outage did not affect the country, explained information security expert Alexey Lukatsky. Against the backdrop of sanctions, most Russian companies were disconnected from Microsoft cloud services, which was the reason for the absence of problems with the software, noted Deputy Head of the Garda Group of Companies Rustem Khairutdinov.