Video summary

"A Tale of Two Cultures: Can Agentic Coding Close the Gap?" plenary by Ken Goldberg at ICRA 2026

Main summary

Key takeaways

Educational

Main Ideas and Lessons

  • “Two cultures” in robotics/AI

    • Model-based robotics (“gofi”): rooted in classical robotics principles—modular components, explicit models, interpretable algorithms, clear interfaces, and analyzable metrics.
    • Model-free robotics (“VLA” end-to-end learning): modern foundation-model approaches (e.g., vision-language-action, diffusion, transformers) that learn policies directly from data. These often excel across varied perceptual-to-action pipelines but can struggle with distribution shifts.

  • A key technical bottleneck: the “robot data gap”

    • Robotics requires learning over very high-dimensional state spaces (e.g., humanoid arms/hands with dozens of degrees of freedom).
    • Unlike language/vision, robotics lacks abundant internet-scale labeled training data.
    • Quantitative claim: robotics data is orders of magnitude smaller than what’s used to train LLMs; one estimate suggests an enormous time to “read” LLM training data at typical human reading speed.
    • But data alone isn’t the whole story:
      • Example contrast: even if Tesla collects far more driving data than Waymo, Waymo can perform better due to strong engineering and modular design (filters, PIDs, mathematical models, system integration).

  • Why “good old-fashioned engineering” still matters

    • Classical robotic pipelines (perception → grasp → motion planning) can be extremely effective and reliable for specific tasks.
    • The talk argues for reducing dogma: VLAs are promising, but engineered approaches can still outperform as conditions vary.

  • A concrete case study: DexNet → Ambi Robotics → real production success

    • DexNet (grasping)
      • Build a dataset not of images only, but of 3D object models + grasps.
      • Account for uncertainty in object pose/shape/gripper pose using a probabilistic graphical model and Monte Carlo integration.
      • Use depth sensors (shape-based sensing) and explicitly model sensor noise (including specular/transparent issues).
      • Train a network to predict probability of successful grasps from a noisy observation + candidate grasp.
      • Control approach: sample many grasps, score them with the network, select the best.
      • Reported performance: state-of-the-art-ish around 2017–2018, and surprisingly robust on bin-clearing with novel object sets.
    • Jeff Bezos demo
      • Demonstration with mixed objects; a “shoe” test becomes a standout moment when the system successfully grasps it.
    • Ambi Sort (package sorting in production)
      • Depth scanning → robot picks → reads zip code → places into destination bins.
      • Reinforces data + modular engineering, especially motion planning.
      • Motion planning difficulty: fast IK isn’t “solved” in tight, time-critical, contact-rich environments.
      • Jacobi Robotics: motion planning improved by warm-starting planning using neural networks.
    • Data avalanche / production data flywheel
      • Production generates continuous logs of successes/failures, enabling continual improvement.
      • Example milestone: 100 million packages sorted and long-term operational logging.
      • Metaphor evolves:
        • Not just a “flywheel” (stabilization), but more like an “avalanche” (growing data and performance).

  • Generalist robotics is attractive, but current ROI/work remains small

    • The talk cites an argument that total useful paid work is currently a rounding error relative to hype and investment.

  • “Variational automation” as a realistic target

    • Robotics in production often isn’t fully general, but is repetitive with variation:
      • Same task family (e.g., sort packages, make coffee, plug/unplug cables) with changing initial conditions.
    • Robotics progress is framed as specialist → gradually broader, rather than “everything at once.”

Methodologies / Instruction-Like Content (Detailed)

A) DexNet grasp-probability estimation (from uncertainty modeling to control)

  1. Build a grasp dataset (3D CAD models + candidate grasps)

    • Represent objects with CAD geometry.
    • Represent grippers (e.g., parallel jaw gripper).
    • Enumerate grasp candidates (e.g., contact facets/points).

  2. Model grasp success probabilistically

    • Treat uncertain variables as random variables:
      • object pose uncertainty
      • gripper pose uncertainty
      • shape uncertainty
      • center-of-mass uncertainty
    • Treat the grasp as uncertain because real contact outcomes differ from nominal geometry.

  3. Compute success probability

    • For a nominal grasp, perturb uncertain factors.
    • Use Monte Carlo integration over those perturbations.
    • Output: probability of success (example numbers given: ~22% brittle grasp; ~92% robust grasp).

  4. Incorporate sensing noise

    • Use depth sensing because geometry matters more than color/texture.
    • Model depth sensor noise by synthesizing what the CAD model would look like under the sensor, then adding realistic noise artifacts.
    • Data points: (noisy observation Y, grasp parameters) → success probability label.

  5. Train a neural network

    • Use a large network (on the order of ~million parameters as described).
    • Train on a dataset containing many positive/negative grasps (high vs low probability).
    • Validate on held-out examples to verify generalization.

  6. Use the network in a grasp-selection loop

    • Observe a pile/bin via depth images.
    • Sample many candidate grasps.
    • Score each rapidly using the network.
    • Select the grasp maximizing estimated probability of success.

  7. Demonstrate deployment robustness

    • Test on objects not in training.
    • Extend iteratively to new object types and setups.

B) Production data-driven improvement (“data avalanche”)

  • Deploy robots to perform real work.
  • Log every pick attempt (successes and failures).
  • Use operational data for:
    • predictive maintenance (e.g., blocked suction cup detection/alerts)
    • learning improvements (including complex object categories like deformable bags)
  • Train generative/learning models on real operational data.
  • Iterate:
    • improved robot performance → higher usage → more data → further improvement

C) Graph as Policy (GAP): combining agentic coding with modular robotics structure

Motivation

  • VLAs struggle under increased variation; reliability and compositional structure are needed.
  • Agentic coding can provide interpretability (code/structure) but is hard at scale.

Core shift

  • Move from code-as-policy (LLM outputs code) to graph-as-policy:
    • the agent generates a computation graph (nodes + edges), not monolithic code.

Inputs formalized for “variational automation tasks”

  • language description of the task
  • environment information (geometry, coordinate frames)
  • robot/sensor details
  • object set / object class information
  • state space (subset of robot/object states relevant to the task)
  • belief space (possible encountered states/distribution)
  • reward function (how success is evaluated)

Graph synthesis + execution architecture

  • A “harness” uses LLM power to:
    • segment the problem into modules
    • assemble an initial computation graph
    • parameterize it using geometry/object information
  • Self-learning loop:
    • execute the graph
    • measure performance
    • modify/improve the graph (described as genetics/evolution-style iterative optimization)

Deployment

  • Output: an optimal graph (G*).
  • Ship to robot with a lightweight graph executor—no continuous reliance on GPUs/LLMs onboard.

D) Combining model-free and model-based for better VLA performance

  • Use graph-based policy (GAP) to reduce uncertainty/variation in perception/action alignment.
    • Example: position camera and gripper over the object before invoking the VLA (or a “VLA-like policy Pi 0.5”).
  • Benefit:
    • VLA performance improves substantially (described as roughly doubling/tripling success rate in their experiments).
  • Additionally:
    • successful production systems can generate new data to further train model-free methods.

High-Level Trajectory of the Talk

  1. Childhood inspiration → bridging science/art → robotics/automation “two cultures”
  2. Establish model-based vs model-free contrast
  3. Argue the “robot data gap” and question “data alone”
  4. Provide the DexNet/Ambi Robotics production story
  5. Introduce “data avalanche” and “variational automation”
  6. Introduce Graph as Policy as a constructive synthesis:
    • interpretable modular structure
    • agentic generation of robotics computation graphs
    • self-improvement and good performance across variations
  7. Conclude that model-based and model-free might be converging

Speakers / Sources Featured (Explicitly Named)

  • Ken Goldberg (speaker)
  • Al (referenced as “Al”; not fully identified in subtitles)
  • Roger Taylor (quoted/attributed analogy)
  • Jensen Huang (quoted; “$50 trillion” claim attributed in talk)
  • Michael Black (uncertainty/data analysis basis)
  • Matt Mason (Ken’s advisor; supportive of robot-art efforts)
  • Fei-Fei Li (ImageNet; systematic data collection referenced)
  • Jeff Mahler (PhD student; worked on collecting 3D grasp datasets)
  • Jeff Bezos (Amazon; hosted a demonstration)
  • Ty Brady (Bezos’s assistant who tried the shoe)
  • Shankar Sastry
  • Ruzena Bajcsy
  • Seth Hutchinson
  • Katie Kwan
  • Russ Tedrake
  • Jeff Ichnowski (warm-starting motion planning; Jacobi Robotics)
  • Leslie Kaelbling (flywheel vs avalanche metaphor discussion)
  • Pokit (mentioned; full identity unclear in subtitles)
  • York Gang (co-founder of Dyna; ROI referenced)
  • Elon Musk (generalist robotics context mentioned)
  • Claude (agentic coding systems mentioned; brand referenced)
  • NVIDIA (Nemo Guard referenced; brand referenced)
  • Tip-Top (research system referenced)
  • Cosmos (world model referenced)
  • Grok (world model referenced)
  • Waymo and Tesla (autonomous driving data/engineering comparisons)
  • Bosch (industrial collaboration on crate washing)
  • Libero (benchmark referenced)
  • IEEE Robotics and Automation Society (context for robotics vs automation distinction)
  • DeepMind, Stanford, Princeton (referenced as authors of related generality/variation work; individuals not named)
  • Eric Chen (student who tried Rust; supports “graph structure helps” idea)
  • ROS (as a robotics computation graph framework)
  • Nemo Guard (referenced by context; brand/platform mention)

Question Panelists and Questions (Speakers/Sources)

  • Subtitles include questions posed to Ken Goldberg without clearly identifying questioners by name. Questions include:
    • question about Jepa / architecture replacing VLAs
    • question about verification / automating correctness checking
    • question about mechanically intelligent/adaptive structures and where that fits in closing the data gap
    • question about physical understanding in LLM coding agents / physics knowledge in graphs
  • Moderating/closing remarks reference:
    • a panel on sustainability (no named panelist)

Speakers/Sources List (As Featured)

Ken Goldberg; Al; Roger Taylor; Jensen Huang; Michael Black; Matt Mason; Fei-Fei Li; Jeff Mahler; Jeff Bezos; Ty Brady; Shankar Sastry; Ruzena Bajcsy; Seth Hutchinson; Katie Kwan; Russ Tedrake; Jeff Ichnowski; Leslie Kaelbling; Pokit; Dinesh Jayaraman; Tomas Lozano-Perez; York Gang; Elon Musk; Claude; NVIDIA; Tip-Top; Cosmos; Grok; Waymo; Tesla; Bosch; Libero; DeepMind; Stanford; Princeton; Eric Chen; ROS; Nemo Guard.

Original video