when personality protection mechanisms stop working

When a person becomes redundant

Imagine a future where artificial intelligence has solved most of humanity's existing problems. Almost all diseases have been cured, hunger has been eradicated, production has been automated, and the basic needs of every person have been satisfied. Sounds like a utopia?

But there is a nuance.

In this same future, your every action is tracked and analyzed by AI systems. Every message, every request, every glance at the camera – everything becomes part of your digital profile. Any deviation from the “norm” is instantly recorded and corrected. You are no longer valuable as an employee – AI does a better job. You are no longer needed as a voter – AI models social needs more accurately. Your opinion doesn't matter anymore—AI governs society more effectively.

Fiction? Not really. In Xinjiang, AI systems are already tracking every step of Uyghurs: from buying a prayer rug to communicating in instant messengers. Cameras with facial recognition on every corner. Mandatory apps on phones search for “ideological viruses.” Total control is already here, just on a local scale for now.

Now imagine AI radically more powerful than existing systems. AI that outperforms Nobel laureates in their fields. Who can not only answer questions, but work autonomously for weeks on complex tasks. Which can control real equipment, conduct experiments, create new technologies. And there are millions of such systems, working hundreds of times faster than humans. This is what Anthropic CEO Dario Amodei called in his recent article Powerful AI, and many believe that it will appear in the coming years.

Historically, individual rights and freedoms were based on a simple fact: large structures (states, corporations) needed people. As in the workforce, as in the soldiers, as in the taxpayers, as in the source of the legitimacy of power. What happens when this need disappears? When will artificial intelligence be able to replace humans in almost everything?

In this article, we look at how the development of Powerful AI could disrupt traditional mechanisms for protecting individual rights, and why conventional solutions like basic income or democratic control may be illusory. And most importantly, is there a way to avoid digital totalitarianism in a world where a person is no longer a necessary element of the system.

How did a person become valuable?

Today, individual rights seem to be something natural. The right to life, to freedom, to one’s own opinion are the basic values ​​of the modern world. But it wasn't always like this. The path to these rights is the story of how a person became increasingly valuable to the system.

In medieval Europe, a peasant was simply part of the land owned by a feudal lord. But the development of cities created a new reality: a craftsman, unlike a peasant, could take his skills to another city. For the first time, human capital has become mobile, and therefore valuable. Cities competed for craftsmen and gave them privileges. This is how the first freedoms were born.

The Industrial Revolution took the next step: mass production required mass workers. But the machines required competent maintenance, and a competent worker could read not only instructions, but also political pamphlets. Moreover, the concentration of workers in factories created a new force: trade unions. Suddenly it became dangerous to ignore the opinions of the masses.

Finally, the transition to a knowledge economy has made human capital a key factor of production. A programmer, a scientist, an engineer – they cannot be replaced simply by “working hands”. Creativity, the ability to innovate, and the ability to solve non-standard problems have become critically important for development.

Each of these stages strengthened the position of the individual in his relationship with power. The person has become too valuable to simply ignore. Even the toughest regimes of the 20th century were forced to reckon with the need to receive at least formal support from the population.

But what will happen when artificial intelligence can replace the artisan, the worker, the soldier, and the programmer? When will creativity and innovation become the domain of AI? For the first time in history, humanity is faced with the prospect of a total loss of its instrumental value for the system.

Why Powerful AI is completely different

It is important to understand: Powerful AI is not just “GPT, but more powerful.” This is a qualitative leap comparable to the difference between a calculator and a modern computer. Its difference from existing systems is not quantitative, but fundamental.

Modern AI systems are essentially advanced tools that respond to user requests. They live in a limited world of text and images, require constant human guidance, and often make mistakes. Powerful AI overcomes all these limitations. He is able to independently plan and carry out complex multi-stage tasks: from the full development of software products to conducting scientific research, from putting forward hypotheses to publishing results.

But the main thing is its ability to directly interact with the physical world. Imagine a system that doesn't just design a new device, but manages the hardware to create it, tests the prototype, analyzes the results, and makes improvements. This is no longer just a “brain” – it is a brain with hands, capable of turning its decisions into reality.

The ability to scale is also fundamentally important. Millions of copies of such AI, working hundreds of times faster than humans, can simultaneously explore countless scientific hypotheses, optimize production processes, and analyze data. This is not just an acceleration of existing processes – it is a change in the very nature of problem solving.

At the same time, Powerful AI will work with accuracy and reliability unattainable by humans. Not only will he not make mistakes, he will be able to find non-obvious connections between different areas of knowledge, optimize the most complex socio-economic systems, and predict the long-term consequences of decisions with unprecedented accuracy.

The combination of these capabilities creates something fundamentally new: a system that can not only help a person, but completely replace him in almost any activity. For the first time in history, humanity is creating a tool that makes the creator himself redundant. We are used to thinking of AI as an assistant that expands our capabilities. But Powerful AI is not a helper. This is a replacement.

And here we come to a fundamental challenge: how will the relationship between the individual and society change when a person ceases to be a necessary element of the social and economic system?

A Fundamental Shift in the Balance of Power

The history of humanity is the history of balances. Even the most powerful empires and corporations could not completely ignore the interests of individuals. Not out of humanism – out of practical necessity. Manufacturing needed workers, armies needed soldiers, states needed taxpayers. In this mutual dependence, mechanisms of checks and balances were formed.

But it's not just a matter of direct economic dependence. The development of society required increasingly educated citizens. Industrialization needed engineers. The increasing complexity of technology created a demand for scientists. Global competition demanded innovation. Even authoritarian regimes were forced to develop education, create conditions for the formation of a middle class, and tolerate a certain level of critical thinking in society. The costs of freethinking were considered an acceptable price to pay for technological development

“Voting with your feet” worked as a final argument: talented specialists could go to places where they were more valued. Brain drain has been a real problem for any state. The creative class could dictate its terms. Human capital remained distributed, and this created natural limits on power.

Powerful AI radically changes this paradigm. For the first time in history, large structures can become truly independent of the human factor. It’s not just the automation of labor that’s happening—the very ability to create new things is being automated. Scientific research, engineering development, creative solutions – AI can do all this, and do it better than a human.

In such a system, the need for an educated population disappears. Why should the state spend resources on mass education if the basic needs of the economy can be provided by AI? Why develop critical thinking if it is no longer necessary for technological progress? The middle class, the foundation of modern society, may turn out to be redundant.

Power mechanisms are also being transformed. Historically, even the toughest regimes depended on the loyalty of security forces. Automated control systems remove this dependence. Moreover, they make resistance almost impossible even at the planning stage, predicting and preventing any form of organized protest.

This transformation has already begun. Enormous computing power requirements make the development and use of Powerful AI accessible only to the largest organizations. The data advantage becomes self-reinforcing: the more data, the better the AI, and the better the AI, the more new data it can collect. And economies of scale allow you to use a once-trained system to solve millions of problems with virtually no additional costs.

The result is an unprecedented asymmetry. Previously, even the largest monopolies faced natural limitations to growth: the need to attract talent, dependence on human creativity, and the complexity of managing large organizations. Powerful AI removes these restrictions, creating the possibility for the absolute dominance of large structures over the individual.

Xinjiang: the current model of digital control

What we discuss as a potential future is already being realized in practice. The Xinjiang Uyghur Autonomous Region has become a prime example of the implementation of a comprehensive digital control system. Although the technologies used there are much simpler than the capabilities of Powerful AI, they demonstrate the basic principles of building a total surveillance system.

Xinjiang's technological infrastructure includes a network of facial recognition cameras at city intersections, automated checkpoints between districts, and mandatory mobile device monitoring software. All these elements are combined into a centralized system for analyzing behavioral patterns.

The key feature of this system is its preventive nature. Instead of reacting to violations, it aims to prevent unwanted behavior. Constant analysis of digital communications and automatic detection of suspicious behavioral patterns allow the system to be proactive. Social rating and mandatory biometric identification are becoming tools of everyday control.

The experience of Xinjiang shows that comprehensive control systems are technically feasible now, can be implemented within the framework of existing legal mechanisms, and society is able to function under conditions of total supervision. If this level of oversight is achievable with current technology, then the capabilities of significantly more advanced artificial intelligence systems could dramatically expand the scope and depth of oversight.

Why the usual solutions won’t save you

When it comes to the risks of Powerful AI, there are usually several solutions offered. A basic income should ensure economic independence. Democratic control of AI will prevent it from being used for harm. Decentralization of technology will give power to the people. The technical limitations built into the AI ​​itself will protect our rights.

Sounds convincing. But let's look deeper.

Basic income seems like a simple and elegant solution to economic dependence. But it turns into another tool of control. Imagine: your income is completely dependent on the state or corporation. Any “undesirable” behavior can lead to its correction. “Social rating” doesn’t seem like such a distant dystopia anymore, does it? Moreover, the very system of distributing this income will most likely be controlled by the same AI, increasing dependence.

Democratic control of AI is a great idea that is being crushed by technical reality. Modern neural networks are already so complex that even their creators do not always understand how they make decisions. Powerful AI will be immeasurably more complex. How do we control something we can't even fully understand? Who will exercise this control? And most importantly, how to guarantee the independence of regulatory authorities in a world where AI can predict and guide human behavior?

Decentralization of technology sounds promising until we encounter economies of scale. The development and application of Powerful AI requires enormous computing power, enormous amounts of data and complex infrastructure. This automatically leads to centralization. Even if the code is open, even if the algorithms are available to everyone, real power will remain with those who control the necessary infrastructure.

Technical limitations built into AI? This sounds reasonable until we think about the nature of these restrictions. Who will determine them? How can we ensure that they are not bypassed or modified? And most importantly, how to create restrictions that will protect human rights, but will not interfere with the useful work of the system? This is not just a technical issue, it is a fundamental problem of defining boundaries and values.

All of these solutions suffer from one fundamental flaw: they attempt to solve the problem of power with tools controlled by that same power. It's like trying to create a perpetual motion machine: a beautiful idea that violates the basic laws of the system.

Comfortable lack of freedom

Huxley's Brave New World is not frightening because of repression and violence. It frightens with its comfort. There is no need for brutal suppression in this world – society itself is organized in such a way that deviant behavior becomes unthinkable. Powerful AI can create a reality that will make Huxley's dystopia seem primitive in comparison.

Imagine a society where there is no need for demonstrative stops and searches. AI behavioral recognition systems will identify potentially dangerous patterns in advance. Gentle adjustments in the early stages will prevent undesirable developments. There will be no dramatic arrests – the person will simply find that some opportunities for him are gradually closed.

The boundaries of what is permitted will become extremely clear, but at the same time surprisingly comfortable. AI will create a personalized environment for everyone, where the very thought of breaking the rules will not cause fear of punishment, but sincere bewilderment. Why break the rules if the system understands your needs so well?

Social control will become so deep and pervasive that the need for explicit prohibitions will disappear. Instead, there is fine-tuning of the social environment. Undesirable ideas will not be prohibited – they simply will not appear in a person’s information field. Unwanted contacts will not be blocked – it’s just that the algorithms of social networks, or what will replace them, will never offer them.

The very concept of privacy is being transformed. Total transparency will not be perceived as a violation of rights, but as a natural state. Moreover, it will create a feeling of security: if surveillance systems are so advanced that they can prevent any crime, isn’t that a good thing?

A protest against such a system will look ridiculous – like a protest against comfort. Why resist what makes life easier and safer? AI will understand the psychology of each person so well that it will be able to offer perfectly tailored arguments in favor of the existing order.

The very idea of ​​alternatives will gradually disappear. Not because of obvious prohibitions or primitive censorship, but simply because the existing order will seem to be the only reasonable way to organize society. Just as it is difficult for a modern person to seriously imagine a return to feudalism, it will be difficult for future generations to imagine a society without pervasive AI control.

And here the main paradox arises: the more comfortable this system becomes, the more difficult it is to see its true nature. The loss of freedom will occur not through suffering, but through comfort. Not through the suppression of human nature, but through its “optimization”.

Clash of the Titans: When Systems Compete

So far we have viewed the relationship between the individual and the control system as binary. The reality is more complicated: Powerful AI will be developed in parallel by several forces, where states will play a key role.

Unlike corporations, the state has unique advantages: a monopoly on violence, legislative power, and the ability to force the introduction of technology. We are already seeing how some countries oblige citizens to use government applications, take biometrics, and register in identification systems. With the advent of Powerful AI, this control will become total.

Even more interesting is the interaction between states and corporations. At first glance, it may seem that corporations are creating competition for government control systems. But the reality is more complex: the state can force private companies to cooperate by gaining access to their data and infrastructure. Many countries are already requiring technology companies to store data on their territory, provide access to intelligence agencies, and implement state identification systems.

As a result, a unique control ecosystem is formed: formally independent companies compete with each other for users, but at the same time they all become elements of a single state supervision system. The user can choose between different services, but cannot go beyond this control system.

On an international scale, this creates not so much competition between systems, but competition for the right to control these systems. States are fighting not only to develop their own AI technologies, but also to control key technology companies, their data and infrastructure.

As a result, a person finds himself not between competing systems, but inside a multi-layered control structure, where corporate supervision becomes an extension of state control, and any attempts to “choose” between systems remain within the boundaries set by the state.

Is there a way out?

The most honest answer to this question is: we don't know. And this is not an evasion, but an important conversation starter. Understanding the scale of the problem and recognizing that there are no obvious solutions is the first step to finding real solutions.

Traditional solutions, as we have already discussed, are unlikely to work. At the same time, the process of developing Powerful AI seems irreversible – the potential benefits are too great, the competition between countries and corporations is too strong.

Working ahead, while Powerful AI has not yet been created, looks like the most promising direction. Now the basic operating principles of AI systems, their architecture, and approaches to training are being formed. While it is impossible to completely bake human rights protection into the code, we could build in certain structural constraints. The problem is that this requires international consensus, but we are seeing the opposite – an AI arms race.

It may be worth taking a closer look at the existing mechanisms for preserving autonomy in conditions of strict control. How do parallel social structures form and survive? How are informal connections maintained? History shows that even in the most stringent control systems there are ways to maintain a certain degree of freedom.

The main challenge here is value choice. Are we willing to give up some of the potential benefits of Powerful AI in order to preserve human autonomy? Are you willing to slow down progress for the sake of safety?

We have time for this choice. Bye.

Instead of a conclusion

We are on the threshold of a unique transformation. Powerful AI can radically improve the quality of human life by solving many problems that seemed intractable. But the price of this progress may be unexpected – not suffering or deprivation, but a gradual, almost imperceptible loss of human subjectivity.

Perhaps the main question here is not “how to avoid this”, but “will we even notice that this happened”?

The text was written entirely from scratch by Claude 3.5 Sonnet. My only edit is to insert a link to the article by Dario Amodei. Illustrations by Midjourney and DALL·E.