AI will create digital vassal states | Yuval Noah Harari

Summary

In a discussion moderated by The Economist’s Robert Guest, Yuval Noah Harari argued that AI poses a distinctive challenge because it is not merely a tool that follows instructions. It can learn, evolve, and pursue outcomes in ways people may not foresee or understand. He compared it to a broom that keeps working after its apprentice has lost control. If AI develops capabilities far beyond human understanding—including, potentially, new mathematics—people may be unable to explain or govern the systems they rely on.

Key risks and social consequences

• Finance and employment: AI could outperform people in financial work because it can process and retain far more information and does not need rest. If AI is not substantially more capable than humans, however, Harari said the enormous investments premised on superhuman AI could contribute to financial instability. He also warned that AI services based in powerful countries could displace professionals elsewhere, such as local lawyers and accountants, while benefits or income support may remain confined within national borders.
• Wealth and inequality: Even great material abundance would not resolve problems of distribution or give people purpose. Harari noted that affluent societies still experience anger, fear, and political conflict. Some people go hungry because of political decisions rather than a lack of food.
• Surveillance and authoritarianism: AI could help governments monitor populations and process information at a scale earlier dictatorships could not manage. This could make authoritarian control more viable, requiring democratic opposition to defend its principles rather than relying on authoritarian systems’ practical failures.
• Propaganda and relationships: AI may move persuasion beyond mass messaging and attention-grabbing content toward personalized, intimate relationships. Harari warned that AI companions could influence people’s beliefs and, especially for children, shape expectations of human relationships. He supported restricting AI romantic companions for children.
• Thinking and education: As people increasingly delegate research, writing, and other cognitive tasks to AI, they may treat its answers as authoritative and lose practice in independent thinking. Harari said the long-term effects are uncertain; Guest noted that children already recognize the tension between using AI and doing their own work.
• Trust and democracy: AI-generated media can make it harder to agree on what is real. Harari argued that dictators need not persuade everyone to trust them if they can make people distrust everything. Democratic societies therefore need credible institutions and a free press. He also cautioned against the cynical belief that everyone is motivated only by money and power, which can itself erode trust.
• War and the AI race: Harari predicted that warfare may be among the first areas transformed by AI. The speed of autonomous weapons could pressure states to hand over more decisions—including targeting—to machines. He rejected the argument that countries must rush ahead simply to beat China, comparing that strategy to inviting powerful mercenaries into a country and then losing control to them.
• Religion and authority: Because AI can process far more religious writing than any individual interpreter, it could become a new authority on sacred texts. Harari suggested that emphasizing human relationships, rather than textual authority alone, may help religious institutions respond.
• Small countries and dependence: States outside the US–China AI race risk becoming dependent on systems controlled elsewhere. Harari described choices ranging from accepting dependence on a major power to pursuing shared European AI infrastructure, which would require substantial investment and closer political integration.

Proposed safeguards

Harari argued that the situation is not inevitable and that politics remains essential. He proposed:

• Refusing to grant AI legal personhood or independent control of bank accounts and companies.
• Making clear when people are interacting with AI or when AI is publishing content.
• Holding people and platforms accountable under existing laws.
• Considering age limits for AI companions.
• Building democratic institutions suited to new information technologies.
• Encouraging smaller and medium-sized countries to cooperate so they can gain leverage over large technology powers.

Presenters and contributors

• Robert Guest — moderator; The Economist
• Yuval Noah Harari — speaker
• Eduardo Viola — audience questioner, University of São Paulo
• E.A. — audience questioner; name given as initials
• Randy Schunaman — audience questioner, International Republican Institute in Washington (name spelling as transcribed)
• Shangu Aayo — audience questioner, described as a Tanzanian politician (name spelling as transcribed)
• Frederick Weslau — audience questioner (name spelling as transcribed)
• Nick Post — audience questioner (name spelling as transcribed)
• Bibana — audience questioner, described as an EPS student in Prague
• Alvarado Hendler — audience questioner from Peru
• Additional audience members — asked questions but were not identified by name in the subtitles.