Game Theory Can Make AI More Correct and Efficient

Researchers are using ideas from game theory to improve large language models and make them more consistent.

Imagine you have a friend who gives different answers to the same question depending on how you ask it. “What is the capital of Peru?” will get one answer, and “Is Lima the capital of Peru?” will get another. You’re probably a little concerned about your friend’s intelligence, and you’ll almost certainly have a hard time trusting his answers.

This is exactly what happens with many large language models (LLMs), the super-powerful machine learning tools that power ChatGPT and other AI wonders. A generative question, which is open-ended, yields one answer, while a discriminative question, which requires a choice between options, often yields another. “A disconnect occurs when the same question is phrased differently,” he said. Atul Paul Jacoba doctoral student at the Massachusetts Institute of Technology.

To make the language model's responses more consistent—and to make the model more robust overall—Jacob and his colleagues devised a game in which two modes of the model are tasked with finding an answer they can agree on. Called consensus gamethis simple procedure pits the LLM against itself, using game theory tools to improve the accuracy and internal consistency of the model.

“Research examining self-consistency in these models has been very limited,” he said. Shaegan Omidshafiichief scientist at robotics company Field AI. “This paper is one of the first to tackle this problem in a smart and systematic way, by creating a game in which the language model can play with itself.”

“This is really exciting work,” added Ahmad Beirami, a research scientist at Google Research. For decades, language models have generated answers to clues in the same way, he said. “With their new idea of ​​bringing a game to the process, the MIT researchers have introduced a completely different paradigm that could potentially lead to a flood of new applications.”

Making the game work

The new work, which uses games to improve AI, contrasts with past approaches that measured an AI program’s success by its mastery of games. In 1997, for example, IBM’s Deep Blue beat chess grandmaster Garry Kasparov, marking a major milestone for so-called thinking machines. Nineteen years later, Google’s DeepMind program, called AlphaGo won four out of five games against former Go champion Lee Sedol, opening up another arena in which humans no longer dominated. Machines also outperformed humans at checkers, two-player poker, and other “zero-sum” games in which one player’s win invariably means his opponent’s loss.

A much more challenging challenge for AI researchers is the game of Diplomacy, a favorite of politicians like John F. Kennedy and Henry Kissinger. Instead of two opponents, the game features seven players whose motives are difficult to understand. To win, a player must negotiate, making cooperative agreements that any one of them can break at any time. Diplomacy is so difficult that the Meta* team was pleased when it was AI program Cicero demonstrated “human-level performance” over 40 games. Although she did not defeat the world champion, Cicero performed well enough to be in the top 10% of players against human competitors.

During the project, Jacob, a member of the Meta* team, was struck by the fact that Cicero relied on a language model to generate dialogue with other players. He sensed untapped potential. The team’s goal, he says, was “to build a better language model to play this game.” But what if they focused instead on building a better game to improve the performance of large language models?

Coordinated interactions

In 2023, Jacob began studying this issue at MIT, working with Ikan Shen, Gabriele Farina and his scientific supervisor Jacob Andreas on what would become the consensus game. The basic idea arose from imagining a conversation between two people as a cooperative game in which success occurs when the listener understands what the speaker is trying to convey. Specifically, the consensus game is designed to align two systems of a language model—the generator, which processes generative questions, and the discriminator, which processes discriminative questions.

After months of stopping and starting, the team turned this principle into a full-fledged game. First, the generator receives a question. It can come from a person, or from an existing list. For example, “Where was Barack Obama born?” The generator then receives several possible answers, such as Honolulu, Chicago, and Nairobi. Again, these options can come from a person, a list, or a search performed by the language model itself.

But before it answers, the generator is also told whether it should answer the question correctly or incorrectly, depending on the results of a random coin toss.

If the result is heads, the machine tries to answer correctly. The generator sends the original question along with the chosen answer to the discriminator. If the discriminator determines that the generator intentionally sent the correct answer, they each get one point as a kind of incentive.

If the coin lands on tails, the generator sends what it thinks is the wrong answer. If the discriminator decides it was intentionally given the wrong answer, they both get another point. The idea here is to encourage compliance. “It’s like teaching a dog a trick,” Jacob explained. “You give them a treat when they do it right.”

The generator and discriminator also start with some initial “beliefs.” These take the form of probability distributions associated with the various choices. For example, the generator might believe, based on information it has gleaned from the internet, that there is an 80% chance that Obama was born in Honolulu, a 10% chance that he was born in Chicago, a 5% chance that he was born in Nairobi, and a 5% chance that he was born elsewhere. The discriminator might start with a different distribution. While the two “players” are still rewarded for reaching an agreement, they also get a penalty for deviating too far from their initial beliefs. This arrangement encourages the players to incorporate their knowledge of the world—again, gleaned from the internet—into their answers, which should make the model more accurate. Without something like this, they might agree on a completely wrong answer, like Delhi, but still score points.

Eventually, the generator and discriminator begin to agree more, as they reach what’s called a Nash equilibrium. This is perhaps the central concept in game theory. It represents a kind of balance in the game—a point at which no player can improve their personal performance by changing strategies. For example, in a game of rock-paper-scissors, players do best when they choose each of the three options exactly one-third of the time, and they will invariably do worse with any other tactic.

In a consensus game, this could play out in a number of ways. The discriminator might notice that it gets a point for saying “true” every time the generator sends the word “Honolulu” for Obama’s birthplace. The generator and discriminator would learn, after repeated play, that they would be rewarded for continuing to do so, and no one would be motivated to do anything else. This consensus is one of many possible examples of a Nash equilibrium for this question. The MIT group also relied on a modified form of Nash equilibrium that incorporates the players’ prior beliefs and helps keep their answers close to reality.

The net effect, the researchers found, is that a language model that plays the game becomes more accurate and more likely to give the same answer, no matter how the question is asked. To test the effects of the consensus game, the team tried a set of standard questions on a variety of medium-sized language models with 7 to 13 billion parameters. These models consistently got higher percentages of correct answers than models that didn’t play the game, even much larger ones with 540 billion parameters. The game also improved the internal consistency of the model.

In principle, any LLM could benefit from playing against itself, and 1,000 rounds would take just a few milliseconds on a standard laptop. “A nice benefit of the general approach,” Omidshafii said, “is that the process is computationally very lightweight, requiring no training or modification of the underlying language model.”

Playing games with language

Following this initial success, Jacob is now exploring other ways to incorporate game theory into LLM research. Preliminary results have shown that an already strong LLM can improve even further by playing another game — tentatively called an ensemble game — with an arbitrary number of smaller models. The main LLM will have at least one smaller model acting as an ally and at least one smaller model acting as an adversary. If the main LLM is asked to name the president of the United States, it scores a point whenever it chooses the same answer as its ally, or whenever it chooses a different answer than its adversary. Not only can these interactions with much smaller models improve LLM performance, tests show, but they can do so without additional training or parameter changes.

And that's just the beginning. Because many situations can be viewed as games, the tools of game theory can be applied to a variety of real-world situations, he said. Jan Hempa research scientist at Google DeepMind. In article from February 2024 For years, he and his colleagues have focused on negotiation scenarios that require more complex exchanges than simple questions and answers. “The main goal of this project is to make language models more strategic,” he said.

One example he discussed at an academic conference is the process of reviewing a paper for acceptance by a journal or conference, especially after the initial submission has received a harsh review. Given that language models assign probabilities to different responses, researchers can build game trees, similar to those developed for poker games, that map the available options and their possible consequences. “Once you do that, you can start calculating Nash equilibria and then ranking a bunch of refutations,” Gemp said. The model essentially tells you: Here’s what we think you should say in response.

With the benefit of game theory, language models will be able to handle even more complex interactions, rather than being limited to question-and-answer problems. “The big gains in the future will be in longer conversations,” Andreas said. “The next step is to have AI interact with a human, not just another language model.”

Jacob sees DeepMind’s work as complementary to consensus and ensemble games. “At a high level, both of these methods combine language models and game theory,” he said, even if the goals are somewhat different. While Gemp’s group translates everyday situations into a game format to help make strategic decisions, Jacob says, “We’re using what we know about game theory to improve language models for general problems.”

Right now, these efforts are “two branches of the same tree,” Jacob said — two different ways to improve the functioning of language models. “My vision is that in a year or two, these two branches will converge.”

*recognized as an extremist organization in Russia

Translation author @arielf

UFO arrived and left a promo code here for our blog readers:
-15% on any VDS order (except for the Warming up tariff) — HABRFIRSTVDS.