This Is How NVIDIA's AI Mercedes Drives in Real Traffic

Summary

At GTC Berlin, David Bombal rides in a Mercedes-Benz CLA running a development version of NVIDIA’s driving software. The car handles much of a real city route—including cyclists, pedestrians, parked cars, construction, narrow streets, unusual junctions, and unpredictable drivers—while a human driver remains responsible and ready to intervene.

How the driving system works

• Alpamayo is NVIDIA’s end-to-end vision-language driving model. It uses images from six of the car’s cameras to interpret the scene and propose a driving trajectory. NVIDIA says it is trained on human driving examples to produce natural, smooth decisions.
• Halos is a separate, conventional safety stack, described as human-written rules and components. It receives feeds from all 10 cameras and data from five radars to build a “world model” and calculate its own trajectory.
• A trajectory arbiter compares the two plans. When they agree and the proposed path is safe, the AI model can drive. If the plans conflict—or the AI’s trajectory risks a collision, violating a signal, or leaving the route—the safety system can block it or take over.
• The NVIDIA representative says the Halos stack rarely needs to drive during the demonstration. It mainly acts as a safety guardrail and can enforce local rules, such as Germany’s prohibition on turning right at a red light.

The software runs locally in the car on a liquid-cooled NVIDIA DRIVE Orin computer. The described sensor setup includes 10 cameras, five radars, and 12 ultrasonic sensors. The representative says the broader Hyperion architecture is designed to be adapted to different vehicles, with tuning for sensor positions, vehicle size, and handling.

Supervision and driving performance

The demonstration is a Level 2 driver-assistance system, not an unattended self-driving car. The driver must watch the road, remain ready to steer, and take responsibility. The car can handle steering, acceleration, braking, turns, and lane changes; driver-monitoring features check attention and presence.

The ride is presented as smooth and cautious. The system responds to pedestrians approaching crossings, gives way on narrow roads, and handles parked cars and construction. The driver also demonstrates taking control to steer around an obstacle. Bombal says he was initially nervous but was impressed by how the car managed unusual situations. Both speakers note that its cautious adherence to rules may feel slow to aggressive drivers.

The NVIDIA representative characterizes Levels 3 and 4 as allowing progressively less driver involvement, with Level 4 limited to a defined area, and Level 5 as fully autonomous anywhere. The CLA shown remains Level 2.

Training, localization, and testing

NVIDIA describes a training process that combines general pretraining data with driving data collected by its test fleet. The representative cites roughly 10 million internet video clips for pretraining and about one million hours of fine-tuning autonomous-vehicle data. The model used in the Berlin demonstration includes around 25% European driving data. NVIDIA says it is the same model used in the US, rather than a Berlin-only version.

Localization also involves adapting the rule-based safety stack. Examples discussed include where traffic lights are positioned, how pedestrian signals are distinguished from vehicle signals, and country-specific traffic rules. The team also had to align the two systems’ interpretations of German junctions and right-of-way rules.

For rare situations, NVIDIA uses real-world fleet data as well as Cosmos simulation to create and test scenarios that may be difficult to encounter often—for example, unusual weather or specific road hazards. Models go through offline regression and simulation testing before being tried on roads. The representative says the development team generates about seven new models per day, though only some progress to road testing and eventual deployment.

Open models and vehicle partners

The production-oriented software running in the CLA is described as a refined development build. NVIDIA also offers an Alpamayo research model on Hugging Face for download. The representative says students and researchers can experiment with it and fine-tune it for projects such as small vehicles or university research, though they need a way to connect it to a vehicle’s controllers.

The representative names Volvo, Hyundai Motor Group, Lucid, and Volkswagen Auto Group as partners using NVIDIA’s Hyperion architecture. He says the architecture can support different vehicle types and powertrains, but requires adaptation for factors such as vehicle dimensions, sensor placement, and differences in acceleration and braking response. The system is described as camera-and-radar-based, without lidar or dependence on high-definition maps.

Main speakers and sources

• David Bombal — Interviewer and passenger, discussing the ride and asking about the system.
• Arman, an NVIDIA representative — Explains Alpamayo, Halos, the vehicle hardware, training, testing, and partner ecosystem.
• The Mercedes test driver — Demonstrates the supervised Level 2 system and takes control when needed.