Video summary

Yann LeCun's $1B Bet Against LLMs [Part 2]

Main summary

Key takeaways

Educational

Main ideas / lessons

  • Core question of the video: Yann LeCun argues that agentic robotics systems need world models—models that can predict consequences of actions—and he views many current Vision-Language-Action (VLA) approaches as fundamentally limited.
  • JEPA as an alternative to mainstream multimodal generative modeling:
    • Mainstream stacks typically build VLMs (vision-language models) by combining vision encoders + LLMs trained with generative next-token objectives.
    • JEPA-style training can be applied at multiple levels of that stack, potentially producing representations that are more efficient and better suited for planning because they learn predictive structure rather than purely caption/next-token matching.
  • Progression through an “alternative stack” (video → language → robots):
    1. Video-based JEPA encoder (V-JEPA-2) replacing a CLIP-like vision encoder inside VLMs.
    2. Extending JEPA thinking toward vision-language models (VL-JEPA).
    3. Going further to world models + planning for robotics (Le World Model), where predictions in embedding space enable explicit action search/planning.
  • LeCun’s critique of VLA (two main points):
    1. Behavioral cloning / scaling demonstrations is brittle and non-scalable
      • VLA often depends on huge volumes of human demonstrations (or UI imitation).
      • This data collection doesn’t scale across task variations; systems can fail hard when conditions change.
    2. Lack of explicit planning / world modeling
      • VLA maps observations + instructions directly to actions.
      • LeCun argues these systems generally can’t predict consequences ahead of time, so they can’t do reliable planning or safety-constrained search.
  • How JEPA world models address this:
    • A JEPA-trained model can be used to simulate outcomes of actions in a learned latent/embedding space.
    • Planning becomes an optimization/search problem over action sequences rather than purely reactive prediction.

Methodologies / “how it works” (detailed)

1) CLIP-style training (baseline mainstream approach mentioned)

  • Inputs
    • An image → vision encoder
    • A caption → text encoder
  • Training objective
    • For matching image–caption pairs: maximize similarity of embedding vectors
    • For non-matching pairs: minimize similarity
  • After training
    • Encoders can be reused for multimodal systems
    • Image encoder embeddings can condition an LLM to create a VLM

2) V-JEPA-2 (video JEPA) training concept

  • JEPA general form
    • Inputs X (e.g., video frames/clips) → encoded into latent embeddings/matrices
    • A predictor learns to predict the latent representation for Y (e.g., missing future/hidden parts)
  • V-JEPA-2 specific idea
    • Self-supervised learning
    • Corrupt video clips by removing patches
    • Feed corrupted + uncorrupted versions through encoders
    • Train predictor to predict embeddings of missing patches
  • Big claim
    • Filling in missing parts forces the model to learn the structure/dynamics of what the world is doing in video.

3) Replacing CLIP vision encoders with V-JEPA-2 inside a VLM

  • Swap
    • Traditional: CLIP-like vision encoder → embeddings → LLM
    • JEPA: V-JEPA-2 vision/video encoder → embeddings → same LLM
  • Goal
    • Test whether a vision-only, non-language-trained encoder can align well enough to help a language model
  • Reported result
    • V-JEPA-2 encoder improves performance on video understanding benchmarks, including cases where captions are not directly supervising the video encoder.

4) VL-JEPA: applying JEPA logic to the full vision-language stack (not generative by default)

  • Architecture mapping
    • Standard VLM:
      • Vision/video embedding + prompt → LLM generates text token-by-token
    • VL-JEPA:
      • Vision/video embedding + prompt → predictor predicts the embedding of the target output text (not the text itself)
  • Training/efficiency rationale
    • Predicting embedding targets reduces penalty for semantically equivalent wording that differs superficially.
  • Reported efficiency gains (from the Meta team mentioned)
    • Under matched training conditions with the same vision encoder:
      • VL-JEPA learns faster (e.g., higher accuracy after fewer examples)
      • Can outperform significantly larger models on visual QA-type benchmarks using fewer parameters.
  • Inference workarounds
    • Multiple-choice style
      • Predict an answer embedding from image+question
      • Compare it to embeddings of candidate answers
      • Pick the candidate with the most similar embedding
    • Generative-like
      • Train additional text decoders to translate predicted embeddings back into readable text

5) Le World Model (JEPA-based robotics world model + action planning)

  • Example task: PushT
    • Move a T-shaped object to a goal position
    • Actions are (roughly) 2D joystick-like controls of an end effector (up/down/left/right)
  • World model training via JEPA
    • Train from trajectories with images + recorded actions (humans perform the task)
    • Not behavioral cloning:
      • The predictor learns next-state prediction in an embedding sense, conditioned on actions
  • Using the world model for planning
    • If needed for interpretability, train a separate decoder to map predicted embeddings back to images
    • Planning algorithm: Cross-Entropy Method (CEM)

Cross-Entropy Method (CEM) planning steps (as described)

  • Given
    • Starting image (initial state)
    • Goal image
  • Initialization
    • Sample many random action trajectories (e.g., 500 trajectories)
    • Group each trajectory into action chunks (e.g., 5 steps per chunk)
  • Rollout in learned embedding space
    • For each candidate action sequence:
      • Encode current image into embeddings
      • Feed embeddings + candidate actions into predictor repeatedly to simulate future embeddings
  • Scoring
    • Compute goal embedding
    • Measure Euclidean distance between final predicted embedding of each trajectory and goal embedding
    • Lower distance = better trajectory
  • Elite selection + resampling
    • Choose top-performing (“elite”) trajectories (e.g., top 30)
    • Estimate mean and standard deviation of elites
    • Resample new trajectories from that distribution
    • Repeat until convergence to a strong plan
  • Execution
    • Follow the planned action sequence in the real environment
  • Stability limitation
    • Planning depth is limited by embedding drift/instability (on PushT it’s reliable for a limited horizon—about five prediction loops).

6) Hierarchical world models for longer horizons

  • Problem
    • Long-horizon rollouts diverge sooner when the model predicts too much with too much detail.
  • Solution concept
    • Hierarchical planning
      • Lower level: detailed short-term predictions/control
      • Higher level: fewer details, abstract long-term planning
  • Subgoal interface
    • Higher-level predictions provide subgoals for lower-level planning.
  • Example described
    • Extend PushT planning horizon from ~5 steps to ~15 steps using two hierarchy levels.
  • Hope for emergent hierarchies
    • LeCun expects hierarchical representations to “emerge” when trained with the right predictive self-supervised objectives at multiple timescales (similar in spirit to how CNN-like hierarchies emerge).

Sources / speakers / entities featured (as mentioned)

Speakers

  • Yann LeCun
  • (Interviewer / host): narrator of the video (Welch Labs presenter)

Organizations / teams / referenced works & models

  • KiwiCo (video sponsor)
  • Physical Intelligence (robotics company; mentions PIO7)
  • Meta (mentions V-JEPA-2 team and VL-JEPA results)
  • OpenAI (mentions 2021 paper/model CLIP; also references “ChatGPT 5.5”)
  • Google (references RT-2; also “Gemini”)
  • Anthropic (references Claude)
  • Welch Labs (video channel/publisher)
  • Johns Hopkins (mentioned in a KiwiCo classroom study)
  • CEM (Cross-Entropy Method) (planning method used in the Le World Model)

Models / systems named

  • PIO7 (Physical Intelligence; vision-language-action model)
  • V-JEPA-2
  • CLIP
  • VLM (vision-language models)
  • VL-JEPA
  • Le World Model
  • PushT
  • RT-2
  • ChatGPT 5.5, Claude, Gemini
  • CNN (referenced conceptually regarding emergent hierarchy)

Original video