The Future of AI Is Absolutely Incredible (Spencer Huang)

Summary

NVIDIA’s Spencer Huang discusses recent robotics tools and why building physical AI is more difficult than training language models.

Announcements and developer tools

• Isaac Lab with Newton updates: Integrates the Newton physics framework into Isaac Lab as NVIDIA moves from PhysX. The update is intended to help developers scale robotics work and use Newton.
• DGX Station blueprints: Provide starting points for training robot policies on DGX Station desktop workstations.
• Agile One SN reference platform: Built with Agile and integrated with NVIDIA Thor. It includes a preconfigured software stack, including Groot, and ships with robot hands. The platform is intended to reduce setup time so developers can start working on applications sooner.

Why physical AI is challenging

Language models can learn from abundant text, images, and recordings. Robotics lacks comparable data capturing how objects and bodies physically interact. Robots also need to handle bidirectional physical coupling: their actions affect the world, and the world pushes back.

Training data can come from:

• Teleoperation, which records robot-specific actions but can be difficult because operators often lack force feedback.
• Human video and motion data, including egocentric hand footage and body-pose or human–object interaction data. This data often needs to be retargeted to robot bodies with different joint layouts.
• Simulation, which can generate large quantities of experience, though simulated data must be combined carefully with real data.

Huang describes this as a co-training and data-mixture problem. Selecting useful proportions of real and simulated data is still partly an art, and models must work across many environments and types of physical dynamics.

Combining conventional and neural simulation

The proposed approach combines:

• Conventional physics simulators, which provide grounded physics but can be laborious to build and may offer limited visual variety.
• Neural simulators or world models, such as Cosmos, which can generate varied environments and observations but may produce physically inconsistent results if they are not grounded in reliable data.

A hybrid setup could use conventional simulation for physical behavior and neural simulation to vary appearance, lighting, materials, and environments. Neural simulators can also generate training data, serve as evaluation environments, and eventually act as part of the policy itself in world action models. Better real-world and conventional-simulation data can, in turn, improve neural simulators.

Robot control and evaluation

• Sonic is described as a roughly 42-million-parameter, open-weights model focused on whole-body control and tracking human motion. It can support diverse body morphologies but requires substantial demonstration data. Huang presents it as a lower-level controller that can work alongside a higher-level VLA such as Groot.
• Whole-body control lets a robot coordinate its upper and lower body—for example, walking while manipulating an object. Robots also need fast, reactive control for balance and immediate responses, alongside higher-level reasoning about longer tasks.
• Isaac Lab Arena evaluates policies in defined scenarios using task metrics. Breaking tasks into subtasks helps identify where a policy fails and what factors, such as lighting or collisions, may be responsible. This can support failure-based learning.
• Sim-to-real is an end-to-end challenge, not just a final transfer step. It involves data collection, data generation, training, robot configuration, evaluation, and deployment. Huang notes that success in simulation alone does not guarantee success on a real robot; tuning for the target hardware and quantifying simulator accuracy are important.

Policies can also learn to compensate for some hardware problems, such as a damaged or locked joint, though doing so may put additional stress on the robot.

Longer-term outlook

Huang expects faster simulation and training to give robots vastly more “experience” than they could acquire in real time. Smaller models could also enable adaptation directly on robots rather than relying entirely on cloud training.

Hard tasks for robots still include dexterous manipulation, tool use, assembly, and sub-millimeter precision. These require suitable data, policies, and hardware.

Huang expects both broadly capable models for everyday tasks and frontier models for harder or specialized problems to remain useful. The interview also mentions NVIDIA robots delivering popcorn as an example of robots beginning to operate in practical environments.

Reviews, guides, and tutorials

No formal product review or step-by-step tutorial is presented. The interview provides a technical overview of NVIDIA’s robotics announcements, training methods, simulation, and open challenges.

Main speakers and sources

• Spencer Huang — NVIDIA; discusses the robotics announcements and physical AI.
• Karoly Zsolnai-Fehér — Two Minute Papers; interviewer.
• Lambda — Sponsor message at the end, describing GPU cloud services for training, inference, and reproducing AI research.