Teaching a humanoid robot to play cornhole with Gemini 4 Argon

Experiment Summary

The experiment tested whether Gemini 4 could teach a humanoid robot to play cornhole by learning from a video of a person throwing a beanbag.

The system tracked the thrower’s arm motion and release, then mapped that movement to the robot’s joints and hands. Using the robot’s two cameras, it estimated the board’s 3D position and created a physics simulation of the robot and board.

The robot practiced in simulation, adjusting its stance and swing to clear its hip and produce a smooth throw. Once it moved to the physical robot, it reviewed where each beanbag landed and adjusted its release point by a few centimeters. The demonstration reports that the robot made seven shots in a row.

Process

The narration outlines a four-step workflow:

1. Observe the human throw and map the movement to the robot.
2. Locate the board and model the physics.
3. Practice and tune the throw in simulation.
4. Review real-world results and refine the action.

The central approach is an iterative video-to-simulation-to-real workflow: learn from a human example, test in simulation, inspect mistakes, and refine the action on the physical robot.

Speakers and Sources

An unnamed Google DeepMind researcher narrates the experiment. Another passage presents the process in the robot’s first-person voice.