One Brain, Any Robot: Skild AI's Skild Brain Explained | NVIDIA AI Podcast Ep. 295

Main technological ideas: Skild’s “omni-brain” for robotics

Robotics as a data problem

Unlike language and vision, robotics lacks large-scale, broadly available “robot data” (there’s no equivalent of an “internet of robot data”). Skild’s argument is to train in a most general fashion, so that each deployment helps improve the brain for future scenarios.

Omni-bodied / universal brain concept

Skild is building a general-purpose “universal brain” that can run across:

• Different robot form factors (e.g., humanoids, dog-like robots, robotic arms)
• Different tasks

This is framed as analogous to how ChatGPT is a general model for language.

General → specialize pipeline (horizontal platform)

Traditional robotics is described as vertical: hardware/software tailored to a single domain. Skild proposes a horizontal model that can be fine-tuned across multiple verticals, so data from one domain can help cover “corner cases” in others.

---

Why this matters: “corner cases” and scaling deployment

Existing robotics systems may reach ~80–90% performance, but physical-world corner cases block full automation and require humans to handle edge situations.

Skild’s thesis: corner cases from one vertical become central cases in another. Therefore, wider/general training plus cross-domain data improves robustness.

---

Data strategy (tutorial/guide-style breakdown): videos + simulation + robot/teleop

Skild uses three complementary data sources:

1. Robot data via teleoperation

   • Provides the richest signal (sensor readings, motor commands).
   • Hard to scale because it requires both a robot and human control (teleoperation).

2. Video data

   • Highly scalable and diverse (collected across regions/countries).
   • Less “rich” than robot sensor/action data—forces and precise actions aren’t fully known.

3. Simulation data

   • Extremely scalable (can generate huge numbers of scenarios).
   • Can measure forces precisely, but suffers from the sim-to-real gap.

Training approach

• Pretrain with video + simulation for scalable foundation learning.
• Post-train with smaller amounts of real-world task data to close the sim-to-real gap and improve precision/robustness for deployment.
• Videos alone are considered insufficient (analogy: watching basketball videos doesn’t make you able to dunk).

---

Deployment process (“process of building/testing/deploying”)

Skild frames robotics deployment as more complex than language model deployment:

• There’s no straightforward “prompt and everyone uses it.”
• Deployment takes time and must be engineered carefully.

Workflow for a new task

• If the task is familiar (e.g., common manipulation/mobility), the brain can be used off-the-shelf.
• For new, significantly different tasks (e.g., assembling a GPU on a conveyor belt), the process is:
  1. Collect data for a few days on the robot (or use simulation if assets exist)
  2. Post-train using domain-specific data
  3. Deploy the updated model to the robot

Data flywheel concept

• Deployments generate “specialist” behaviors across many environments.
• Data is fed back into the shared omni-brain.
• This reduces the marginal data needed for future tasks.

They describe a progression:

• factories/warehouses → semi-structured service environments (e.g., hospitals, hotels, grocery stores) → potentially consumer/home robots.

---

Testing and safety pipeline (explicit evaluation rubric)

Testing is broken into three layers:

1. Task-driven metrics

   • Accuracy + speed/time-to-complete
   • Example: busbar placement

2. Generalization metrics

   • Robustness to unexpected variations (e.g., objects moved/added, lighting changes)

3. Safety guardrails

   • Must prevent hazardous/unsafe behavior
   • Example: if camera input is broken/cut, safety logic should stop/limit actions.

---

NVIDIA technologies used (specific tools mentioned)

• Simulation/physics

  • NVIDIA simulation stack such as Isaac Sim
  • Earlier references include PhysX and Isaac Gym
  • They also mention co-developing better physics solvers (Newton mentioned) with NVIDIA, with possible open-sourcing.

• Video model collaboration / data augmentation

  • Cosmos
  • Using generative video models to augment data (creating variations per data point)

• Compute platform / edge inference

  • Real-time reaction is critical for robotics
  • Emphasis on on-device/edge compute and partnering for inference hardware

---

Product/roadmap focus

• Near-term focus: quickly converting the general model into specialized systems deployable at scale with small fine-tuning in days.
• Motivation: build momentum early despite the longer setup time required for the general data flywheel.
• Claimed major challenge: not only algorithms, but orchestrating deployment at scale, which they say has not been done before.

---

Main speakers / sources

• Noah Kravitz (host, NVIDIA AI Podcast)
• Deepak Pathak (Skild; Carnegie Mellon professor background mentioned)
• Abhinav Gupta (Skild; professor background mentioned)