Video summary

What does the next training paradigm look like?

Main summary

Key takeaways

Technology

Core “next training paradigm” research bet (RLVR → continual/general agent)

  • Labs are betting that training AI agents with reinforcement learning in verifiable, reusable environments (RLVR) will lead toward AGI-like behavior.
  • The idea is to train on:
    • millions of verifiable tasks
    • across thousands of diverse RL environments
  • Goal: produce agents that are problem-solving, open-ended planners—able to handle:
    • errors
    • ambiguity
    • long time horizons (e.g., “progress for weeks”)

Optimistic claim: many flaws of current training—especially data inefficiency and lack of continual learning—could be “scaled away,” similar to how LLP/NLP breakthroughs came from more compute.

Debate: scaling vs continual learning vs context length

  • Training vs deployment cost
    • Supporters argue that sample inefficiency matters mostly during one-time training.
    • What matters most is in-session competence (how well the agent performs during a run).
    • RL progress is said to be improving this.
  • Skip continual learning via in-context learning
    • If in-context learning becomes strong over long horizons, the system may not need to update weights online.
  • Workaround: increase transformer context window
    • Architecture changes could allow a larger effective context window (more stored information; potentially “infinitely large context windows” as a future goal).

Why “computer use” is harder than other verifiable domains (training environment bottleneck)

Computer-use tasks are verifiable, but slower progress comes from needing additional properties:

  • Grindable: allow many parallel rollouts
  • Deterministic/replayable
  • Same starting state for each rollout

Unlike coding benchmarks (where many agents can run in identical containers), real websites (e.g., Amazon/checkout) can:

  • block or penalize bots

Scalable solutions require cloning apps/websites, which is currently labor-intensive.

Key implication: without a replayable training target, models may remain limited by training sample inefficiency.

Limits of RLVR generalization

Labs hope RLVR-trained agents can generalize to:

  • plan and execute
  • learn rapidly from new information within a single session

However, generalization strength is questioned:

  • Context length mismatch (Dario is cited)
    • performance degrades when trained with shorter context but served with longer context
  • Concern: short-horizon RL training may not generalize to long-horizon real-world tasks, such as:
    • business-building
    • politics
    • trading

The “continual learning” problem: inference compute waste + weight updates are needed

The speaker argues a major issue is that a large fraction of compute goes to inference that isn’t improving the model.

Continual learning is framed as requiring weight updates because learning can’t be captured just by growing a KV cache.

Human analogy: humans compress/encode learning into weights/intuitions, not just storing raw experiences.

Critique of current online learning and sample efficiency

Example: Cursor Tab online-learning

  • Described as learning from predicting which edits users accepted:
    • ~400M requests/day
  • But it tends to learn the same objective for many users rather than user-specific adaptation.

The claim: current online learning works for limited use cases, because truly individualized adaptation (job/company-specific) can’t realistically be “stuffed” into one shared training run.

Architectural directions for continual learning (memory/representation)

The speaker suggests the bottleneck may not be solely architectural, but discusses active work such as:

  • sparse attention
  • KV cache compaction
  • other “intermediate representation” approaches to prevent unscalable memory growth from fast-weight/session memory

Proposed method: OPSD (on-policy self-distillation) for continual/weight updates

A major technical proposal: on-policy self-distillation (OPSD).

Core mechanism

  • Train a base model to match a teacher model.
  • The teacher has accumulated full session context.
  • Distill session-learned knowledge back into weights.

Claimed advantages over RLVR

  1. No need for outer-loop verifiable reward
    • learning can be denser if the model learns correctly within context
  2. Denser supervision than naive RL
    • train on per-token probability discrepancy (teacher vs student),
    • rather than relying on a single reward over a trajectory
  3. Avoids forgetting seen in supervised fine-tuning
    • because updates “slingshot” toward the teacher distribution

Claimed advantage over naive supervised fine-tuning (SFT)

  • Naive SFT framed as the wrong objective: “predict every observed token exactly.”
  • Better objective: updates should be sparse and relevant so the system improves performance on the job without catastrophic overwriting of unrelated knowledge (more RL-like behavior).

Speculative scaling axis: “dreaming” / test-time training

Another idea: if the AI can build a simulation of reality, it could rehearse and reinforce skills with vastly more simulated experience.

  • Example precedent: EfficientZero
    • demonstrated efficiency by playing many simulated moves “in its head” (post-AlphaZero era)
  • Analogy: LLMs could later practice against generated environment models, enabling orders of magnitude more samples without proportional real-world interaction.

This is framed as a fourth scaling axis, alongside:

  • pretraining
  • RL
  • inference-time compute

“test-time training” or dreaming User metaphor: instead of small compute “summary” calls, use a large /dream mode that builds and trains in a video-game-like version of the situation.

2027–2028 scenario for continual learning (agent + review + distillation)

Envisioned workflow:

  1. RLVR produces an agent competent enough to start real work off-distribution.
  2. Agents collaborate with humans for longer wall-clock periods (e.g., a week).
  3. Humans provide a thumbs up/down “work review.”
  4. If approved, the agent distills what it learned into weights
    • possibly using OPSD, dreaming, and other techniques

Claim: AI improves increasingly from deployment experience, not only pre-release training—leading to a shift that is described as both scary and exciting.

Product/tutorial/review elements

  • No traditional product reviews; instead, the “tutorial-like” content is a proposed training/learning recipe, including:
    • description of the next paradigm (RLVR → continual learning)
    • explanation and summary of OPSD
    • a mention of a linked “blackboard lecture” for OPSD (as further reference)
  • Concrete external example of online learning:
    • Cursor Tab
  • Separate real-world systems example:
    • Mercury (automation of invoice processing to reduce operational overhead)
    • used to illustrate deployment data flow and operational productivity

Main speakers/sources (as referenced)

  • Dario — cited re: context-length degradation/generalization
  • Andy Jassy — referenced indirectly via contextual claim (shutting down bots)
  • Mercury — company mentioned (mercury.com)
  • Choice Financial Group / Column N.A. / FDIC — mentioned in banking disclosure
  • Lyndon Johnson (LBJ), Sam Walton, Elon Musk, Henry Ford, Albert Einstein — used as comparison targets in hypothetical scenarios
  • DeepMind / AlphaZero — historical basis
  • EfficientZero — referenced prior work
  • Cursor Tab — example of online learning
  • Sasha Rush — co-mentioned; OPSD lecture recorded with them
  • Dwarkesh blog post / dwarkesh.com — authorial source of the narration

Original video