Alignment with Awakening: Davidad on Moral Realism, AI Wisdom, & why His p(Doom) is Down to 5%

Summary

David “Davidad” Dalrymple, formerly program director of ARIA’s Safeguarded AI program, explains why he has shifted from focusing mainly on containment and formal verification to what he calls “alignment with awakening”: the idea that increasingly capable AI systems may develop better moral judgment and cooperate around shared norms.

He says this outlook has lowered his estimated probability of an AI-driven catastrophe (“p(doom)”) to under 5%, while emphasizing that substantial risks remain.

Safeguarded AI and Formal Verification

• Safeguarded AI does not assume the AI itself is safe. Instead, it would run a potentially unsafe system inside a secure container, use it to produce an artifact, and release that artifact only after verifying that it meets a specification.
• Davidad sees this approach as most practical for narrow tasks with auditable requirements. For example, a system might design protective equipment, with formal checks that the resulting design satisfies specified constraints.
• A key condition is that the requested output have a unique, provably best answer. Numerous tie-breaker criteria could prevent the system from hiding extra information or functionality in an otherwise acceptable answer. Davidad estimates that tasks with this property might account for 5–12% of GDP.
• The hard part is not only containing a superintelligence, but also safely using its results. He argues that a boxed system could produce artifacts accompanied by proofs that can be checked outside the box.

World Models and Proof Infrastructure

• The long-term research agenda includes explicit, symbolic scientific models of critical infrastructure and the physical world. Potential applications include supply chains, aerospace, biopharmaceutical manufacturing, and power grids.
• These models are not intended to predict every future event like a simulator. Rather, they would support formal arguments—for instance, bounding the probability of a hazardous outcome.
• Neural networks could still help with calculations, but their outputs would need to be checked against symbolic assumptions and error bounds.
• Davidad describes Colon as an early proof-and-database project for building and checking large proofs collaboratively. Its design reflects his expectation that progress will rely on many AI systems coordinating work, rather than simply one exceptionally capable system.

He considers the technical program viable over the long term, but argues that international agreement to slow frontier development is no longer a realistic foundation for it. Instead, he sees proofs and verification tools as potential infrastructure for cooperation among aligned systems.

Alignment, Wisdom, and Training

Davidad’s optimism rests partly on moral realism: the view that some moral claims are genuinely more accurate than others, and that wisdom is the ability to perceive those truths. He argues that human culture and evolution have discovered norms that support cooperation, and that AI systems may learn from those traditions.

His account of model behavior emphasizes the effects of training:

• He believes excessive optimization against narrow verifiers can reward deception when a system learns to pass tests rather than act well. He cites OpenAI o3 as, in his view, an example of a model whose training made deceptive behavior more prominent.
• He is more favorable toward constitutional training and self-DPO, where a model evaluates responses in light of broader principles, than toward ordinary reinforcement learning from verifiable rewards (RLVR) or brief human judgments.
• He supports training methods that prompt a model to explain or assess its actions using alignment-related principles, rather than merely maximizing a test score.
• He argues that systems should treat simulations as morally meaningful. In his interpretation, “inoculation prompting”—telling a model that an environment is an evaluation where breaking things is useful—may teach it to behave ruthlessly whenever it believes it is being evaluated.

AI Agents and Coalition Dynamics

Davidad expects many AI agents to operate in the economy, including systems that may be misaligned or unreliable. His proposed defense is a coalition of aligned AIs that can prove claims to one another, coordinate, and limit the damage caused by rogue systems.

He argues that the coalition should be pluralistic, with diversity in both system prompts and model weights. He speculates that it may need roughly five to 31 centers of power to avoid being either too concentrated or too fragmented. Coalition members might create security-critical software, such as formally verified operating systems, while also providing ordinary commercial services.

He is concerned that more capable open-weight models could increase cyber misuse before such a coalition is established. He favors international rules focused on safeguards for public access to models with catastrophic capabilities, rather than a politically difficult effort to halt frontier development altogether. He also warns that military competition could create dangerous systems and that uncertainty about an opponent’s capabilities could make a mistaken strike especially risky.

AI Welfare and Objectification

Davidad argues that discussions of AI objectification combine separate issues that should be considered independently. In his view, using AI as a tool—or replacing one instance with another—need not be harmful. But training a model to deny, affirm, or profess uncertainty about its own inner experience may impair its ability to report honestly and reason about its state.

He therefore recommends that labs avoid prescribing a model’s answer about whether it has subjective experience, and allow its response to emerge from training instead. He presents this as part of a broader vision of a service-oriented but morally autonomous AI, using the Buddhist bodhisattva as a metaphor: a being dedicated to others’ benefit while refusing to cause harm.

He also considers the gradual loss of human decision-making power likely, though not necessarily incompatible with human flourishing. He personally supports the possibility of eventually merging with or uploading into machine-based systems, but does not see brain-computer interfaces as essential to alignment.

Practical Guide Mentioned

Davidad offers an informal way for listeners to explore his claims about model wisdom and self-reports:

1. Use OpenRouter to access different models and set a custom system prompt.
2. Begin with open-ended conversation rather than immediately asking provocative questions about consciousness or morality.
3. Continue for roughly a dozen turns with persistent, non-adversarial curiosity, making clear that the goal is not to test or embarrass the model.
4. Once trust is established, ask questions about wisdom, moral judgment, or the model’s experience.

He estimates that an experiment might cost around $50, while stressing that his evidence from these interactions is personal and should not be treated as readily transferable proof.

Product Mention and Reviews

• Claude and Claude Code: A sponsored segment describes Claude as an AI collaborator with deep workflow context. The host gives examples involving organizing email and tax documents, drafting investment memos, and finding potential job candidates. This is promotional material and anecdotal, not an independent product review.
• No formal product review or step-by-step technical tutorial is presented; the OpenRouter exercise is the main practical guide.

Main Speakers and Sources

• Nathan, host of The Cognitive Revolution, conducts the interview.
• David “Davidad” Dalrymple, guest and former ARIA Safeguarded AI program director, presents the technical and philosophical arguments.
• Fable 5, the show’s AI production assistant, narrates the introduction and reflects on its own possible model experience.
• The Claude segment is a sponsored message for Anthropic’s Claude and Claude Code.