Андрей Карпати - Мы вызываем призраков, а не создаём живых существ

High-level summary

Andrey Karpathy (former Tesla Autopilot lead) gives a pragmatic, wide-ranging discussion of current AI (LLMs, agents, reinforcement learning), how progress has actually happened, what’s missing, and how education should adapt. His core metaphor:

Today’s systems are “ghosts” — disembodied imitators trained on internet data — not “animals,” which are organisms shaped by evolution and lifelong embodied learning.

He expects steady, decade-scale progress rather than an abrupt one-year takeover.

Key technological concepts and analyses

Pretraining vs. evolution

• Pretraining on internet text yields powerful, knowledge-rich models but results in “ghosts”: systems that imitate human-produced data rather than embody compact, evolved learning algorithms.
• Evolution (and biological learning) likely encodes compact meta-learning/lifelong-learning algorithms. Pretraining accumulates knowledge and instills some meta-learning, but it also encourages over-reliance on memorized internet facts.

Agents, LLMs, and the “Decade of Agents”

• Early RL agents (Atari, OpenAI Universe) struggled due to sparse rewards and inefficient exploration.
• Modern agent progress depends on strong pre-trained representations (LLMs and multimodal models).
• Missing pieces for robust agents include:
  • true multimodality,
  • computer-use and tooling skills,
  • continuous lifelong learning,
  • persistent memory and cross-session distillation,
  • richer tooling ecosystems.

Reinforcement learning (RL) critique

• RL has low information density: sparse, end-of-trajectory rewards force noisy credit assignment — “sucking supervision through a straw.”
• Process supervision (feedback on intermediate steps) is conceptually stronger but hard to automate. LM-based judges can be adversarially fooled; allocating robust partial credit is difficult.
• Promising directions: simulation, synthetic tasks, reflection/grading of intermediate steps, and scalable stepwise feedback — but these are not solved at scale.

Context window, KV cache, and working-memory analogy

• Model weights encode a highly compressed, vague memory of pretraining data.
• The context window / KV cache functions like working memory: much higher bits-per-token availability during inference.
• This explains why models perform better when given source text in-context versus relying on memorized facts.

Transformers as cortical analog / architectural trends

• Transformer architecture behaves like a general-purpose cortical tissue; reasoning/planning is analogous to prefrontal processing.
• Transformers are likely to remain central but will evolve (sparse attention, sparse MLPs, improved kernels, longer contexts, and better memory/distillation mechanisms).

Model scaling, distillation, and the “cognitive core”

• Data quality matters: internet data is noisy. Better-curated corpora could enable much smaller, more efficient “cognitive cores.”
• Karpathy conjectures a compact cognitive core (order of ~1B parameters) might suffice for strong reasoning if fed high-quality distilled data — though this is open to debate.
• The trend: very large models followed by more compact, smarter distilled models. Progress requires improvements across data, compute, algorithms, and infrastructure.

Collapse, synthetic data, and entropy

• LLM self-generated outputs can exhibit mode collapse (low diversity); self-training on synthetic data risks exacerbating this.
• Preventing collapse requires preserving entropy/diversity in synthetic data — an unresolved research challenge.
• Biological analogues (e.g., dreaming) may inspire methods that create off-distribution scenarios to avoid overfitting.

Multi-agent systems, culture, and self-play

• Under-explored ideas include:
  1. Building a persistent “culture” (shared, writable artifacts that LMs can read and edit).
  2. Scalable self-play/task-generation loops where models create progressively harder tasks for peers.
• These could bootstrap capabilities but are not yet convincingly implemented at scale.

Code models, auto-completion, and agents

• Current code models excel at auto-completion and common templated patterns (high throughput for routine code), but they struggle with unique, repository-specific, intellectually demanding engineering work.
• Agents can automate repetitive template work effectively; designing bespoke systems usually requires human-led design and careful integration.
• Practical workflow: use LMs as completion assistants or for lower-risk refactor tasks, not as full replacements for deep engineering.

Autopilot analogy and the “march of nines”

• Demos are easy; production-grade reliability is hard. Each incremental “nine” in reliability (90% → 99% → 99.9%) typically demands comparable, large engineering effort.
• Real-world autonomy rollouts are gradual, often rely on human teleoperation/overrides, and face economic plus regulatory constraints.

Economic and societal impact / takeoff views

• Karpathy views AI as a continuation of long-term automation (gradual diffusion) rather than a discontinuous GDP spike.
• Expect gradual redistribution of tasks (an “autonomy slider”) rather than an instantaneous explosion.
• Early automation targets: tightly scoped, repetitive, fully digital tasks (e.g., call centers, some programming tasks).
• Complex sociotechnical roles (e.g., nuanced medical advice) are harder to automate and may persist longer.
• Risks include diminishing societal understanding/control as systems become more autonomous and competitive pressure pushing premature automation.

Product features, guides, tutorials, and educational projects

NanoChat

• An ~8,000-line open repository implementing an end-to-end chatbot pipeline (training from scratch, full stack).
• Created as an educational artifact; Karpathy advises learners to rebuild it from scratch (don’t copy-paste) to learn deeply.
• Practical tip: use two monitors, place the repo on the right, and reimplement modules to internalize design choices. Autocompletion helps for routine parts but the architect must lead.

Micrograd

• A ~100-line Python repo implementing autograd (forward/reverse) to teach the core math behind neural network training.
• Presented as a minimal ramp to understand gradient-based learning; most other work is efficiency engineering.

CS231N and LM101N

• Karpathy references his CS231N course and is building LM101N / NanoChat as parts of a larger LM 101N course and an educational project (“Eureka”).

Eureka (education project)

• An initiative to build high-quality, tutor-driven “ramps to knowledge” that emulate the best human tutors.
• Emphasizes tutors’ ability to diagnose student level and provide the right “challenge zone.”
• Approach: combine human teachers, curated artifacts (NanoChat, micrograd), and LLM assistants; automate routine tutoring incrementally while retaining human design quality initially.

Practical learning advice

• Learn by building: write and reimplement code; don’t just read blog posts or slide decks.
• Re-implement educational repos (e.g., NanoChat, micrograd) rather than copy-paste to discover hidden complexities.
• Use LLMs as assistants (autocompletion, drafts), not as surrogates for deep understanding.
• Teach by exposing first-order simplifications first, then add higher-order corrections; use problems where students attempt solutions before guided discussion.

Industry tools & companies referenced

• LLMs/Agents: OpenAI GPT series (GPT-5 Pro mentioned), Claude, Codex
• Self-driving: Tesla Autopilot, Waymo
• Data/annotation: LabelBox (example of high-quality human-in-the-loop trajectories and thought-process annotations)
• Research references: AlexNet (AlekNet), DQN/Deep RL on Atari, Rich Sutton, Geoffrey Hinton

Research and engineering gaps / suggested directions

• Long-term continuous learning: session-level experiences must be consolidated into weights (a distillation/sleep phase) and paired with external persistent user memory.
• Better process supervision: develop robust partial-credit mechanisms and improve LM-judges to resist adversarial reward gaming (GAN-style defenses and robust evaluation).
• Maintain diversity in synthetic data: avoid collapsed distributions when self-generating training data.
• Multi-agent culture & self-play loops: build shared writable artifacts and ecosystems for adversarial/cooperative task generation.
• Data quality improvements: curate higher-signal pretraining corpora to reduce unnecessary memorization.

Practical takeaways / recommendations

• To learn LLM/agent stacks: clone NanoChat but reimplement it yourself to learn the end-to-end pipeline; study micrograd to understand backprop from first principles.
• Use code autocompletion for throughput and to learn unfamiliar APIs, but expect human engineering for novel architectures and project-specific design.
• Expect gradual deployment with long engineering tails for high reliability. Invest in better data, improved evaluation/judging, and robust partial-feedback algorithms to advance RL and process supervision.

Main speakers and sources

• Andrey Karpathy — primary speaker (technical analyses, product and education projects: NanoChat, Micrograd, Eureka; reflections on Tesla Autopilot and AI research/practice).
• Interviewer / podcast host — asks questions and runs the interview (unnamed).
• Mentioned entities and references: OpenAI (GPT family), Claude, Codex, Waymo, Tesla, LabelBox, Geoffrey Hinton, Rich Sutton, and various academic papers on in-context learning, sparse attention, RL, and self-play.