Is It Just a Fancy Dictionary, or Is Something Emerging?

Summary

The video explores emergence: how interactions among simple components can produce complex patterns or abilities that no one explicitly designed.

Emergence in nature and simple systems

• Conway’s Game of Life uses four simple rules on a grid of living and dead cells. Despite having no rule for movement, certain patterns—such as the glider—move across the grid while retaining their shape.
• Flocks can coordinate without a leader. In Craig Reynolds’s 1987 computer simulation, digital birds follow simple rules: avoid collisions, align with nearby birds, and stay near the group. Real starlings are also described as attending to roughly six or seven nearby birds.
• Ant colonies exhibit coordinated behavior even though individual ants make decisions based on local interactions.
• The brain offers another example: individual neurons respond to signals, while large networks of neurons give rise to complex capacities. The video cites an estimate of about 86 billion neurons in the human brain.
• Physicist Philip Anderson’s idea, “More is different,” captures the principle that new properties can arise at higher levels of organization and cannot always be inferred from the parts alone.

How language models learn

A language model is trained to predict the next token in a text. The text is divided into tokens, converted into numbers, and processed by the model. When a prediction is wrong, the model’s internal parameters are adjusted slightly. This cycle—guess, check, nudge—is repeated at enormous scale.

The video argues that abilities such as grammar, arithmetic, recalling facts, and following a story are not separately programmed into the model. Instead, they can develop as the model gets better at predicting text.

GPT-3 demonstrated few-shot learning: it could infer a task from examples in a prompt without changing its parameters. As models grew, some benchmark scores appeared to jump sharply, including scores on arithmetic and other tasks. The prompt “Let’s think step by step” also substantially improved a large model’s performance on one math-word-problem test, while similar prompting did little or even harmed smaller models.

The debate over “emergent abilities”

In 2023, researchers led by Rylan Schaeffer argued that some apparent sudden leaps might be a measurement effect. If a test gives credit only for a completely correct answer, gradual improvement can look like a sudden jump. Partial-credit scoring can reveal steady progress instead.

Their analysis found that many reported abilities appeared under harsh, all-or-nothing scoring, and that rescoring could turn apparent cliffs into slopes. They also demonstrated that scoring choices could create apparent emergence in image models.

The counterargument is that some tasks genuinely require an exact answer. The video’s conclusion is that the debate refined how researchers measure model abilities but did not settle whether all apparent leaps are gradual.

Evidence from inside models

Researchers studied a model trained on 20 million Othello games, represented only as move sequences. The model was not given a board or the game’s rules, yet it made legal moves about 99.99% of the time.

To test whether it represented the board internally, researchers:

• Held out a quarter of possible games and tested the model on them.
• Trained a small probe to read the board state from the model’s internal activity. It recovered about 98 of every 100 squares correctly.
• Edited the model’s internal activity to flip a square and observed that its predicted move changed to match the altered board—even for impossible board positions.

These interventions support the claim that the model both represented and used an internal board-like state. Neel Nanda later observed that the representation was from the player’s perspective—“mine” and “theirs”—rather than simply black and white.

Other examples suggest that models may form internal structure:

• A 2023 study using probes on Llama 2 found an approximate representation of geographic locations, with cities arranged near their real-world locations. The representation was sharper in larger models. The video treats this as weaker evidence than the Othello result.
• In a 2025 study, Anthropic researchers found signs that Claude anticipated possible rhyming words before writing the next line. Switching off or changing an internally active word altered the line the model generated, suggesting a form of planning.

Current picture

The video’s overall claim is that next-token prediction can give rise to internal structures such as a board, a map, or a plan. Researchers still debate how abrupt new abilities are, and the models’ inner workings remain incompletely understood. As of 2026, researchers may sometimes anticipate an ability shortly before it appears, but cannot reliably predict what a future model will gain.

Researchers and sources featured

Named researchers and thinkers

• John Conway — Mathematician who developed the Game of Life.
• Richard Guy — Mathematician who first noticed the glider.
• Craig Reynolds — Computer graphics researcher who demonstrated flocking behavior in 1987.
• Philip Anderson — Physicist associated with the idea “More is different.”
• Ilya Sutskever — OpenAI co-founder, cited on the relationship between next-word prediction and understanding.
• Rylan Schaeffer — Lead author of the Stanford study questioning whether emergent abilities are a mirage.
• Neel Nanda — Researcher who noted the Othello model’s player-perspective representation.

Papers, organizations, and research groups

• “More Is Different” (1972), by Philip Anderson.
• “Language Models Are Few-Shot Learners,” the GPT-3 research paper.
• The 2022 research cataloguing emergent abilities, described as work by researchers from Google and elsewhere.
• “Are Emergent Abilities of Large Language Models a Mirage?” by Schaeffer and two Stanford colleagues.
• The Othello-model research and subsequent probing work; the researchers are not named in the subtitles.
• The 2023 Llama 2 probing study by two MIT researchers; they are not named in the subtitles.
• The 2025 Claude rhyming and planning research by Anthropic researchers; they are not named in the subtitles.
• A major international AI report authored by more than a hundred experts; the report is not named in the subtitles.
• OpenAI, in connection with GPT-2 and GPT-3; NeurIPS, where the Stanford paper received an outstanding paper award.