Chronicles of Facebook’s Biggest Failure, Causes and Serious Implications for the Company
On October 4, users from all over the world noticed that the social network Facebook stopped opening. Moreover, several FB-related services have also stopped working, including Instagram, WhatsApp and others.
After a while, Discord, Twitter, Google, TikTok, Telegram, YouTube, Netflix, Zoom and Viber stopped working partially or completely. Then it seemed to many users that a worldwide Internet apocalypse had begun, which put an end to the existence of the global network. A little later it turned out that not everything is so gloomy. Now everything has calmed down, so it’s time to remember this event and evaluate the causes and consequences of the failure.
How large was the failure?
It has been called the largest since 2008, and it is, and it should be borne in mind that over 13 years the number of users on the social network has increased many times. Accordingly, it is this failure that can be called the largest in the history of the social network. In addition to the services mentioned above, even Oculus virtual reality glasses began to fail – neither the installation of new applications and games, nor social functions didn’t work…
By the way, the failure relatively accurately coincided with the appearance in the sale of data of 1.5 billion users of the social network. The database contained elements such as names, e-mail, id, location and phone numbers of users. True, it later turned out that the drain occurred before the global outage of the social network.
The problems turned out to be so serious that the employees of the social network could not even immediately get into the company’s data center, and the internal network also stopped working normally.
All this chaos lasted for about 6 hours, after which the situation began to gradually return to normal.
What was the reason?
Both freelancers and the Facebook team statedthat the reason for such a large-scale failure is a change in the configuration of the central routers. They are responsible for coordinating traffic between the data centers of the social network.
Inexperienced Employee The company representatives performed an unsuccessful reconfiguration of the routers, which resulted in traffic disruption and other problems. The equipment in question is responsible for BGP sessions and their announcements, plus the company’s internal network. After that, the FB NS servers were no longer available, and the DNS records disappeared as well. Naturally, under such conditions, there can be no question of any normal operation of the social network and its services.
The failure was cumulative, which can be compared to the effect of falling dominoes. One problem caused another, one caused the next, and so on. There are many rumors about the culprit or the culprits of the global network outage, but she did not make an official statement on this matter.
The failure was not promptly eliminated precisely because of the fall of not only the external, but also the internal network of the company. So the team of emergency responders simply did not manage to quickly get to the data centers. There were no riddles, the casket was easy to open – the passes did not work for the employees.
The problems of large services are a very interesting topic, but we have other articles, check it out – we talk about:
→ Small “raspberries” in a large data center
→ Dynamic UPSs in data centers: how we installed Piller CPM300 double conversion
→ Disassembly of a rare beast from Nvidia – DGX A100
The consequences of a failure
The problem with routers affected, first of all, the social network itself. The company lost $ 99.75 million, although the actual figure could be even higher. The fact is that the losses of Facebook were calculated by representatives of Fortune.
The journalists decided to take as a basis the profit of the social network for the second quarter and then use a simple formula to find out how much the company earns per hour of work. The figure turned out to be rather big – about $ 13.3 million per hour. It was multiplied by the number of downtime hours and received the same $ 100 million.
The head of the company, Mark Zukurberg, lost much more. In a matter of hours, his fortune decreased by $ 6 billion, which, however, is not particularly sensitive for him, given that the total capital of the creator of the social network was $ 140 billion at the time before the failure. Other companies also suffered. So, on the day of the shutdown of the Facebook promotion fell by 4.89%, Twitter – by 5.79%, Alphabet – by 2.11%, Amazon – by 2.85%.
However, reputational rather than financial losses were more sensitive for the company. The reliability of Facebook’s infrastructure has been called into question, so the social network will have to do a lot in order to make such situations impossible in the future.
By the way, after the statement that “we fixed everything and this will not happen again” followed by a second failure. On October 8, 2021, Facebook, Instagram and Whatsapp dropped again. At first, users had problems with the feed of photos, and then they could not publish posts or send messages.
Users felt the consequences in full – according to experts, 68% of Russians noted malfunctions in services. The problems themselves affected the work of every sixth Russian.
A couple of hours after the fall of FB, difficulties began to be observed in services not related to the company. The fact is that users began to massively switch to other products, the servers of which were not ready for such an influx. For example, the traffic of the Telegram messenger grew by 17% in just a few hours. The messenger climbed 55 positions at once in the ranking of downloads in the US on the iPhone.
And the fall in social services has caused an active increase in the efficiency of developers’ labor. Experts have calculated that the number of pull requests from Haystack users
Obviously, Facebook’s problems did not end with technical issues. Now, perhaps, the problems are just beginning.
The social network crashed at an extremely unfortunate moment for Facebook. Then lawsuits were filed, notable media outlets including The Wall Street Journal and The New York Times published negative articles.
Users stopped trusting the system, gradually rolling back to other platforms including Signal, Telegram, Clubhouse and Twitter.
The company faces significant losses – after all, it will have to deal with its own advertisers. Representatives of a number of companies have already stated that they are counting on resolving the situation, with compensation for the lost advertising opportunities and compensation for losses.
The experts once again thought about the security issues of the global Internet. There is “foolproofing” here, but it is not very reliable. Without major changes to the infrastructure of the global network, such failures can be repeated. Several teams – and a significant part of the global Internet lies.
The company will have to deal with the US Congress, which does not like the social network itself or its creator. In the near future, congressmen are likely to demand that the company solve all major problems – there were many of them before, but now another has been added.
Be that as it may, but times for FB come relatively difficult. It is difficult to predict how the situation will develop in the future, so we can only wait.