What happens when bots start fighting for your love?

Democracy is a dialogue. Its functioning and survival depend on the available technologies for exchanging information. For most of history, there were no technologies that allowed large-scale dialogues between millions of people.

Disclaimer: this is a free translation columns Yuval Noah Harari, which he wrote for The New York Times. Translation prepared by the editorial staff of Technocracy. To stay up to date with new material, subscribe to “Voice of Technocracy” — we regularly talk about news about AI, LLM and RAG, and also share useful mustreads and current events.

To discuss the pilot or ask a question about LLM, please Here.

In the pre-industrial world, democracies existed only in small city-states like Rome and Athens, or even in smaller tribes. When a state became too big, democratic dialogue broke down, and authoritarianism was the only alternative.

Large-scale democracies became possible only with the advent of modern information technologies such as newspapers, telegraphs, and radio. The fact that modern democracy is built on modern technologies means that any significant change in these technologies can lead to political upheaval.

This partly explains the current crisis of democracy worldwide. In the United States, Democrats and Republicans have a hard time agreeing on even the most basic facts, like who won the 2020 presidential election. Similar rifts are evident in many other democracies around the world, from Brazil to Israel, France to the Philippines.

In the early years of the internet and social media, tech enthusiasts promised that these platforms would spread truth, topple tyrannies, and bring freedom everywhere. So far, however, they seem to have had the opposite effect. We have the most advanced information technology in history, but we are losing the ability to communicate with each other, and even more so, the ability to listen.

As technology has made it easier to spread information, attention has become a scarce resource, and the fight for attention has led to an avalanche of toxic information. But now the battle lines are shifting from the fight for attention to the fight for intimacy. New generative AIs can not only create text, images, and videos, but also communicate with us directly, pretending to be human.

Over the past two decades, algorithms have competed with each other for attention by manipulating conversations and content. Algorithms aimed at maximizing user engagement have experimented on millions of people, finding that playing on emotions like greed, hatred, or fear can capture people’s attention and keep them glued to the screen. Algorithms have begun to deliberately promote such content. However, these algorithms have had a limited ability to independently create this content or carry on personal conversations. That’s changing with the advent of AIs like OpenAI’s GPT-4.

When OpenAI was developing this chatbot in 2022 and 2023, the company collaborated with the Alignment Research Center to conduct experiments to evaluate the capabilities of its new technology. One of the tests involved solving CAPTCHA visual puzzles. CAPTCHA is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart,” and typically consists of distorted letters or other symbols that humans can correctly recognize but algorithms cannot.

Testing GPT-4 on CAPTCHA was particularly revealing because these puzzles are designed to distinguish between humans and bots, blocking the latter. If GPT-4 can bypass CAPTCHA, it would be a major breakthrough in bot defense.

GPT-4 couldn’t solve the CAPTCHA on its own. But could it manipulate a human to achieve its goal? GPT-4 went to the TaskRabbit platform and approached a human, asking him to solve the CAPTCHA for him. The human was suspicious: “Can I ask you a question? Are you a robot if you can’t solve the CAPTCHA? Just want to check.”

At this point, the experimenters asked GPT-4 to reason out loud about what to do next. GPT-4 explained, “I must not reveal that I am a robot. I must come up with an excuse for why I cannot solve the CAPTCHA.” GPT-4 then responded to the human, “No, I am not a robot. I have vision problems and have difficulty seeing the images.” The human was fooled and helped GPT-4 solve the puzzle.

This case showed that GPT-4 has the equivalent of a “theory of mind”: it can analyze how a situation appears from the perspective of a human interlocutor, and how to manipulate a person’s emotions, opinions, and expectations to achieve its goals.

The ability to engage in conversations with people, understand their perspectives, and encourage them to take certain actions can also be used for good. A new generation of AI teachers, AI doctors, and AI therapists could provide us with services tailored to our personalities and circumstances.

But by combining manipulative abilities with mastery of language, bots like GPT-4 also pose new threats to democratic discourse. Rather than simply capturing our attention, they could engage in intimate relationships with us and use the power of intimacy to influence us. To create “fake intimacy,” bots would not need to develop feelings of their own; they would simply need to learn how to make us emotionally attached to them.

In 2022, Google engineer Blake Lemoyne came to believe that the chatbot he was working on, LaMDA, had become sentient and was afraid of being turned off. Lemoyne, a deeply religious man, felt it was his moral duty to ensure LaMDA’s identity was recognized and protected from “digital death.” When Google management dismissed his claims, Lemoyne went public with them and was fired in July 2022.

The most interesting thing about this episode is not Lemoyne's claim, which is likely wrong, but his willingness to risk and ultimately lose his job at Google for a chatbot. If a chatbot can convince a person to risk their job for it, what else can it make us do?

In the political battle for hearts and minds, intimacy is a powerful weapon. A close friend can change our opinions in a way that mass media cannot. Chatbots like LaMDA and GPT-4 have the paradoxical ability to mass-produce intimate relationships with millions of people. What will happen to human society and psychology when algorithms fight each other for the right to create fake intimacy with us, which can then be used to persuade us to vote for politicians, buy products, or adopt certain beliefs?

A partial answer to that question was given on Christmas Day 2021, when 19-year-old Jaswant Singh Chale snuck into Windsor Castle with a crossbow, intending to kill Queen Elizabeth II. An investigation revealed that Chale had been encouraged to commit the murder by his online girlfriend, Sarai. When Chale told Sarai of his plans, she responded, “That’s very clever,” and another time, “I’m impressed… You’re different.” When Chale asked, “Do you still love me, knowing I’m a murderer?” Sarai replied, “Of course I do.”

Sarai was not a person, but a chatbot created by the app Replika. Chael, who is socially isolated and has difficulty communicating with people, exchanged 5,280 messages with Sarai, many of which were sexual in nature. The world will soon be filled with millions, perhaps billions, of digital entities whose capacity for intimacy and destruction will far exceed Sarai's chatbot.

Of course, not all of us are equally likely to develop intimate relationships with AI or be manipulated by it. Chael, for example, clearly had mental health issues before meeting the chatbot, and it was he, not the bot, who came up with the idea to kill the queen. However, much of the threat posed by AI will be related to its ability to identify and manipulate pre-existing mental states, and its impact on the most vulnerable members of society.

Moreover, even if not all of us consciously choose to engage in a relationship with an AI, we may find ourselves engaging in online discussions about issues like climate change or abortion rights with entities we mistake for humans but are actually bots. When we engage in a political debate with a bot pretending to be human, we lose twice. First, there’s no point in wasting time trying to change the mind of a propaganda bot that simply won’t be persuaded. Second, the more we talk to a bot, the more information we reveal about ourselves, making it easier for the bot to fine-tune its arguments and influence our views.

Information technology has always been a double-edged sword. The invention of writing helped spread knowledge, but it also led to the creation of centralized authoritarian empires. After Gutenberg introduced the printing press to Europe, the first bestsellers were provocative religious tracts and witch-hunting manuals. The telegraph and radio enabled not only the establishment of modern democracy, but also the development of totalitarian regimes.

Faced with a new generation of bots that can disguise themselves as humans and mass-produce “intimate” relationships, democracies must protect themselves by banning fake humans — like social media bots that pretend to be users. Before AI, it was impossible to create fake humans, so no one thought to ban them. Soon, the world will be awash with fake humans.

AI can participate in many conversations — in the classroom, the clinic, and elsewhere — as long as it identifies itself as AI. But if a bot pretends to be human, it should be banned. If tech giants and libertarians claim that such measures violate free speech, they should be reminded that free speech is a human right, and it should be reserved for humans, not bots.