Machine learning in game development
In recent years, we have seen an explosive growth in the popularity of online multiplayer games that have captured the hearts of millions of players around the world. As a result, the requirements for game designers are growing exponentially, because players want to see thoughtful mechanics and balance. After all, there is no interest if one strategy is noticeably superior to all others.
When developing gameplay, the balance is usually adjusted according to the following scheme:
Thousands of game games are held with the participation of testers.
Feedback is collected and, based on this, adjustments are made to the game.
Steps 1 and 2 are repeated until the result is satisfactory for both testers and game designers.
This process is not only time consuming, but also imperfect. The more difficult the game, the more likely minor flaws will go unnoticed. When games have many different roles with dozens of interrelated skills, it can be very difficult to achieve the right balance.
Today we are introducing a machine learning engine that helps adjust game balance by training models acting as testers. We will demonstrate the approach with an example of an experimental computer card game. Chimera… We have already shown it as an experimental system for graphics generated by machine learning algorithm… In this test, trained software agents run millions of batches among themselves. From their results, statistics are collected that help game designers improve balance, improve the game and bring it closer to its original intention.
Chimera
We conceived Chimera as an experimental game, in the development of which machine learning will be actively used. For her, we deliberately proposed rules that expand the possibilities and seriously complicate the creation of traditional game AI algorithms.
In Chimera, players from other creatures create chimerathat need to be developed and made stronger. The goal of the game is to defeat the enemy’s chimera. The gameplay includes the main points described below.
Players can use:
creaturesthat attack (damage depends on the indicator attacks) or defend themselves (losing the indicator health);
spellsthat give special effects.
Creatures are summoned in biomes limited capacity, which are placed on the playing field. Each creature has a preferred biome, and it takes permanent damage if it ends up in an unsuitable or overpopulated biome.
At the very beginning, the player receives a chimera embryo, which needs to be developed and made stronger, adding fragments of other creatures. To do this, the player must collect a sufficient number of bond energies from various sources.
The game ends when the player brings the opponent’s chimera to zero health.
Chimera Learning
Chimera is a card game with incomplete information and a large state space, so we thought it would be difficult for the model to learn to play it. Especially considering that we were going to use a relatively simple model. They adopted the approach used in early game agents such as AlphaGowhen convolutional neural network (CNN) is trained to predict the probability of winning from an arbitrary game state. We trained the first model on games with arbitrary moves, and then forced the agent to play against himself, collecting data for training his subsequent iterations. Each time, the quality of the data improved, as did the agent’s playing skills.
We have chosen an encoded image as the game state that is fed to the CNN input. This approach turned out to be more efficient than all procedural agents and other types of neural networks (for example, fully connected ones). The chosen model architecture is compact enough to run on the CPU in a reasonable amount of time. Therefore, we loaded the model weights and ran the agent in real time in the Chimera game client on the platform Unity Barracuda…
Adjusting the Chimera Balance
Our approach helped to simulate many millions more matches than live players could have played in the same time period. After collecting data from the games of the most successful agents, we analyzed the results and found an imbalance between the two game decks we invented.
First, Evasion Link Gen, consisted of spells and creatures with abilities that gave the additional binding energy necessary for the development of the chimera. It also had spells that allowed creatures to dodge attacks. In the deck Damage-Heal there were creatures with different indicators of strength and spells that healed and inflicted minor damage. We thought that these decks would be roughly equivalent, however Evasion Link Gen won 60% of the time when playing against Damage-Heal.
After collecting different indicators for biomes, creatures, spells and the development of chimeras, we immediately saw two things:
The development of the chimera gave a clear advantage – the agent who developed his chimera more than his opponent won in most games. At the same time, the average number of evolutionary stages per game did not meet our expectations. We wanted to make development a key part of the game mechanics and increase the overall average number of evolution stages.
The creature Tirex was too strong. His appearance in the game was closely correlated with victories, and the model always tried to play with Tirex, regardless of the penalties for calling him into an unsuitable or overpopulated biome.
Based on these observations, we have made some changes to the game. To encourage the development of the chimera, we have reduced the amount of bond energy required for evolution from three to one. We’ve also added a cooldown for Tirex: now after any action he had to wait twice as long to perform the next action.
After repeating the game of the model against itself with the updated rules, we noticed that the game changed in the right direction – the average number of evolutionary stages increased, and Tirex ceased to dominate.
Weakened Tirex to reduce the dependency of the deck Evasion Link Gen from an overly strong being. Even so, the win ratio remained at 60/40 instead of 50/50. A closer look at the magazines of individual games showed that they were often conducted without a well-thought-out strategy. After analyzing the data again, we found a number of other areas where changes could be made.
We have increased the starting health of both players, as well as the effectiveness of healing spells. This helped increase the length of the games and diversify the strategies. In particular, it allowed the player with the deck Damage-Heal hold on long enough to take advantage of your treatment strategy. To help players be smart about summoning and placing creatures, we have increased the penalties for using creatures in unsuitable or overpopulated biomes. Finally, we have closed the gap between the strongest and weakest creatures by slightly tweaking the attributes.
With the new adjustments, we now have the final balance scores for the two decks:
Conclusion
It can usually take months of testing to find imbalances in new games. The approach discussed here helped us not only to identify possible balance deficiencies, but also to make adjustments to eliminate them in a matter of days. We found that a relatively simple neural network can provide high efficiency when playing against humans and traditional AI. Such agents can also be used for other purposes, for example, when teaching new players and looking for non-obvious strategies. We hope this work will inspire further exploration of the possibilities of machine learning in game development.
Words of gratitude
This project was implemented with the support of many people. We thank Ryan Poplin, Maxwell Hannaman, Taylor Stiyl, Adam Prince, Michal Todorovich, Xuefang Zhou, Aaron Kammarat, Endip Tour, Trang Le, Erin Hoffman-John and Colin Bosewell. Thanks to everyone who participated in game testing, gave advice on game design, and provided valuable feedback.
