He Found a Demon Inside GPT-3 That Nobody Put There

Overview

Mathematician Matthew Watkins describes how he entered the effective altruism (EA) and AI-alignment community, then discusses his research into unusual tokens in GPT-3 and GPT-3.5. The interview combines technical observations about language models with speculation about their cultural and psychological implications.

AI Alignment and Its Research Community

• Watkins joined an AI-alignment training program in 2021–22, when GPT-3 existed but ChatGPT had not yet made large language models mainstream. He says the program focused on hypothetical advanced AI and gave language models relatively little attention.
• He describes EA as a loose network rather than a single organization. Its emphasis on directing resources toward the greatest measurable good helped fund AI-safety training and research.
• The interview explores the community’s ambitions and tensions. Some members feared advanced AI could threaten humanity, while also hoping safe AI might enable vast future expansion. Watkins questions how far utilitarian calculations should outweigh present-day needs and notes that money, status incentives, and a decentralized structure can create opportunities for abuse.
• The conversation also touches on MATS, Berkeley’s Light Cone community, and the overlap between EA, AI alignment, and the rationalist forum LessWrong.

GPT-3’s “Glitch Tokens”

Watkins’s central story concerns a set of unusual strings—later called glitch tokens—that produced anomalous outputs when entered into GPT-3 and related models.

• One example was “SolidGoldMagikarp.” When prompted to repeat it, the model might instead answer “distribute,” insist that it had said the requested word, or produce other unexpected responses.
• Another token, “petertodd,” elicited a capitalized, hyphen-separated message amounting to “In this world there is no madness, nothing fair.” Watkins says he could not find that sentence elsewhere online.
• Other tokens reportedly prompted refusals, insults, threats, religious-sounding declarations, or incoherent text. At the time, these behaviors attracted attention online and in media coverage.

Watkins and collaborator Jessica Rumbelow investigated how such strings could arise. The explanation they describe is an artifact of tokenization: models process text as token IDs rather than as ordinary letters or words. A tokenizer built using byte-pair encoding (BPE) creates a vocabulary of common words and text fragments. Watkins suggests that rare or oddly formed entries in a vocabulary originally developed for GPT-2 were carried into later models, even though those tokens were not useful or well represented in the training data.

This also relates to early models’ difficulty spelling words or counting letters: a model’s tokens do not necessarily match the characters or words a person sees. Watkins’s examples demonstrate unexpected behavior, but the interview does not establish that the tokens contain a hidden agent or intention.

Watkins’s Interpretation and Its Limits

Watkins describes the Peter Todd outputs as seeming to cluster around disturbing themes and uses Jungian language such as “shadow” to characterize the impression. He suggests that language models may absorb broad narrative patterns and cultural archetypes from their training text. The host and Watkins discuss how models can learn recurring structures in stories and myths, while also acknowledging the danger of pareidolia—seeing meaningful patterns in noise.

The interview presents the “demon” framing as Watkins’s provocative interpretation, not as a demonstrated technical property of GPT-3. It also connects the “petertodd” token to Canadian Bitcoin developer Peter Todd. A later documentary identified Todd as Satoshi Nakamoto; Todd denies being Satoshi, and the interview treats the connection as unresolved.

Book and Other Topics

Watkins’s book, SolidGoldMagikarp: A Descent Into the AI Underworld, expands on the glitch-token research, the surrounding AI and rationalist communities, and his personal experience investigating the models. He says it initially received little media attention, though he had begun receiving positive responses from readers and researchers.

The interview also briefly covers Watkins’s music: free improvisation, a performance combining mathematics and jazz, and a planned ensemble performing 13th-century Spanish Marian music in an improvisational style.

Reviews, guides, or tutorials: No product review, technical guide, or step-by-step tutorial is presented. The discussion is an interview and analysis.

Main Speakers and Sources

• Dr. Josh C. Simmons — Host and interviewer.
• Matthew Watkins — Mathematician, AI researcher, and author of SolidGoldMagikarp; discusses the glitch-token investigation and his views on AI alignment.
• Other referenced sources: Jessica Rumbelow, Watkins’s collaborator on glitch-token research; the interview also refers to online discussions, media coverage, and a later documentary about Satoshi Nakamoto.