Under the hood of Fortnite. How the client-server architecture of Fortnite: Battle Royale works
Almost seven years have passed since the release of Fortnite. The game was an almost immediate commercial success, but the advent of the competitive Battle Royale mode breathed new life into it, turning Fortnite into a true cultural phenomenon. Already in the fall of 2018, the number of players in Fortnite: Battle Royale approached 80 million, and two years later the number of registered users reached 350 million. At the same time, millions of players from different parts of the world fight in the battle royale. How do Fortnite servers handle this kind of load?
Like many other modern multiplayer games,
developed on a popular engine
. In Battle Royale mode, 100 users fight in one game location until only one remains alive. When a player launches Fortnite: Battle Royale on their computer and clicks the Play button, the local client establishes a connection to Epic Games' cloud servers. There, a group is formed, which includes 99 other players, after which a game server is assigned to this group. There is a copy of the Fortnite server application running with the map loaded. The game starts after all players connect, and its action takes place directly on the server, that is, the server determines the state of the multiplayer game at each moment in time.
It should be noted here that the game server does not broadcast the image directly to the user’s device: the client application transmits control directives corresponding to the player’s actions to the server, and data describing changes occurring in the game space is sent in the opposite direction. The image of the game world and the events in it is formed on the local machine, all base maps and skins are also stored there – the client loads only local changes from the outside, for example, the coordinates of weapon caches randomly scattered across the map. In other words, using the information received about the game world and other players, the client application builds the most approximate model of what is currently happening on the server.
To create a feeling of “instantaneity” of events, the game engine on the local machine tries to predict events that will happen on the server in the next few seconds, builds the corresponding picture in advance and loads the necessary sounds – this is called Client-side prediction. Thanks to this function, the player gets the feeling that the action is happening in real time and there are actually no delays. However, if the data transfer speed between the client and the server is low, or the network is overloaded, lags may occur: the player thinks that he hit the opponent, but he has actually already moved in space, and the server did not transmit information about the damage caused to the client. For the same reason, in some games, users may see hit markers where there is no enemy on the screen, or notice that new buildings appear with some delay. At the same time, in order to save resources, the player does not “see” the entire map, but only what is happening directly around him. In the Unreal engine, this function is called Replication Graph. Research conducted using a traffic analyzer Wiresharkshowed that on average the Fortnite client application sends about 50 packets per second to the game server, and receives about 30.
Because players are located in completely different geographic locations and are connected to the internet using different communication channels, they are all actually playing a slightly out of sync version of Fortnite: Battle Royale. These “timing discrepancies” can be measured literally in fractions of a second, but they sometimes determine in multiplayer games which player, for example, shot first. Such delays do not have a serious impact on the gameplay if you are running Fortnite to relax at home after a busy day at work, but they are crucial if we are talking, for example, about a tournament game with a significant prize pool. Here the stability of remote servers, connection speed and ping come first. Amazon Web Services technical evangelist Ian Massingham puts it this way: “With 100 players simultaneously moving, walking, shooting and dying in the same game area, to implement these multiplayer elements I need an infrastructure running in the cloud that will allow me to share the state of the game world with other players. This means that when I shoot a gun in the game, you will see a zombie drop in your copy of the client“
Since the launch of Fortnite, the game's network infrastructure has been based on the Amazon Web Services (AWS) cloud platform. The load on Epic's game servers fluctuates depending on the day of the week and time of day, with the maximum influx of players usually observed during the Christmas holidays. Each month, Epic's servers process two petabytes of data, which is approximately 223,000 DVDs with a capacity of 4.7 GB each. AWS allows you to flexibly manage your available resources, scaling them up or down as needed. It is this scalability that guarantees stable operation of the entire infrastructure as a whole, even under conditions of significant load fluctuations. In addition, the cloud provider's tools allow you to protect game servers from DDoS attacks, which are a significant problem for the gaming industry. As Ben Albon, a leading operations engineer at a British game developer, said in an interview Jagex“DDoS attacks pose a huge risk to online gaming. MMORPGs are much less resistant to failures than other online services. When you and your friends are in a brutal final boss battle that you've been preparing for days for, even a thirty-second glitch is sure to ruin your evening.“
To reduce access time to services and web applications, AWS has 55 geographic availability zones across multiple countries and continents. If one zone fails, the load is dynamically transferred to neighboring ones. Fortnite runs on servers in 24 of Amazon's 55 existing availability zones, another important factor affecting the stability of the gaming ecosystem. One of the Epic employees claimedthat in each specific region the difference between the maximum and minimum peak load on game servers can reach tenfold values. Cloud infrastructure allows you to smooth out such fluctuations. However, when the game reached 3.4 million concurrent users in February 2018, it experienced six major crashes. Epic explained this by saying that it had reached technical limitations in one of its regions, causing the game to crash. Obviously, even clouds have strictly defined resources and limits of capabilities.
Despite the fact that each player is automatically assigned the most suitable game server based on a number of factors: his geographic location, current load and server availability, ping, that is, the period of time during which data packets travel from the computer with the Fortnite client to the game server and back can be quite large. A packet's journey across the network takes an intricate route through many routers and switches, each of which adds its own small delay, so the smaller the physical distance between the client and the game server, the better the ping will be.
In multiplayer battles, it is ping that often determines which player was the first to occupy a wall or a critical area on the map, who fired first, or took possession of a first aid kit. The lower the delay in data transmission over the network, the higher the player’s chances of winning. I remember in the late 90s, I and other young specialists who had just graduated from college and got a job at a research institute played Quake online in Deathmatch mode. Most players then connected to the Internet via dial-up, but at our research institute we had fiber optics, and the game server was accessible via the same broadband channel, since it was located in a nearby administrative building, which had turned into an office center. The “modem players” simply didn’t have the slightest chance against our team. This simple example clearly shows why connection speed to a game server is extremely important. Ping, as they say, decides.
Epic Games publicly displays the public status of its servers: on the company’s website, anyone can find out whether certain Fortnite servers are currently running or not, and when they will next be shut down for maintenance or installation of updates. Fortnite itself shows players technical details about their connection, but this is what they call “information for information.” But Epic Games carefully hides data on the exact location of game servers. Why?
There is a myth among Fortnite fans that somewhere on Earth there are magical places with “zero ping”, in which the speed of data exchange between the client application and the server reaches maximum values. On tournament days, professional players try to come to such cities to stay in hotels with high-quality and high-speed Internet access. However, the exact addresses of the servers, down to the city and street, are kept in the strictest confidence: acknowledging the fact that residents of some localities have a certain advantage in professional-level tournaments over other players will not add points to Epic’s reputation. Not to mention that it poses a certain security risk.
However, Fortnite fan communities have tried to identify such locations. If you believe Estnn, in the United States, game servers are located in seven cities (including Los Angeles, San Francisco and Portland), in Europe there are six locations (including London, Paris, Milan and Helsinki), five in Asia, and one – in Australia, Bahrain and South America (Sao Paulo). This infrastructure should provide players with quick access to the nearest server for a reliable connection to the game, and is likely determined by the total number of Fortnite users in various regions. However, these addresses change from time to time and servers “move” from one data center to another. As for the specific number of servers, it is certainly unknown to anyone except responsible Epic employees, since there can be several such machines in the same location. That said, the same physical servers can serve users of different games: Rocket League, Lego Fortnite, Racing, Festival, but if you're near one of these locations, you'll likely experience low ping on most Epic multiplayer games.
However, the notorious “zero ping” is physically unattainable, even if you play on a local network, since small delays in data transfer will still occur. The only important thing is that in multiplayer your ping is compared with that of your opponents, and the greater this difference, the higher (or, conversely, lower) your advantage. However, when 100 players simultaneously transmit data about their shots and movements to the server, and the server, after processing it, returns information about changes on the map to each player, overload often occurs due to the large volume of data transmitted and analyzed. At such moments, it doesn’t matter how powerful the player’s computer is or how good his ping is – lag can occur on the server itself, and nothing can be done about it.
But an overloaded server is not the only reason why users may experience delays or incorrect game behavior. Another possible reason is packet loss due to an unreliable Internet connection. Many Unreal Engine-based games, including Fortnite, use the protocol by default UDP, a distinctive feature of which is the lack of a guarantee of packet delivery. If you're playing Fortnite on 2.4 Megahertz Wi-Fi and someone nearby turns on a microwave or decides to watch a movie on the online cinema using the same connection, you may have problems. Finally, Fortnite game servers transmit data about changes to the map to players at a maximum frequency of up to 30 packets per second. Therefore, even if the Fortnite client is installed on a modern computer with a huge amount of RAM and a powerful video card connected to a very fast Internet channel, the user will still not receive an overwhelming competitive advantage over other players. Although the game will probably look better on his monitor than, say, on an old laptop.
In other words, Epic Games is doing everything to ensure that the Fortnite gaming infrastructure is as reliable as possible, and uses modern cloud technologies for this. But the reality is that the speed of data exchange and the performance of the game actually depend on a huge number of different factors, many of which are beyond the control of the player himself. However, the Unreal Engine source code is available for study, and by exploring it, players could theoretically find ways to increase their chances of winning in multiplayer. And not just in Fortnite.
The article is supported by the team Serverspace.
Serverspace – a cloud service provider that provides rental virtual servers with Linux and Windows OS in 8 data centers: Russia, Belarus, Kazakhstan, the Netherlands, Turkey, USA, Canada and Brazil. To build an IT infrastructure, the provider also offers: creation of networks, gateways, backups, CDN, DNS services, S3 object storage.
IT infrastructure | Cashback 17% by code HABR
