How NVIDIA Connects 72 GPUs in One Rack

Summary

The interview explains why large-scale AI depends on networking as much as on GPUs. When an AI application runs across thousands of machines, network performance affects how much time GPUs spend computing rather than waiting for one another. Throughput and latency matter, as do jitter and predictability, because machines must coordinate successive stages of work.

How AI systems scale

• Scale up: Connect multiple GPUs within a rack so software can use the rack as a unified compute system. The example discussed is a rack containing 72 GPUs and weighing about two tons.
• Scale out: Connect separate compute systems or nodes so an AI workload can be distributed across them. Unlike a single rack presented as one device, this requires software and algorithms to account for multiple physical systems.
• Scale across: Connect multiple data centers so they can support workloads or move data between sites. Kagan says NVIDIA is developing networking designed to provide low jitter and predictable performance over long distances.

The discussion notes that AI workloads differ from conventional cloud services. Rather than independently serving many applications, a large training job may run as one coordinated application across tens of thousands of nodes. A delayed node can hold up the rest, making a well-balanced combination of computing and networking essential.

Networking technologies

• NVLink connects GPUs for scale-up within a system. Kagan describes it as similar to an on-chip network, rather than a technology intended to link millions of devices.
• InfiniBand and Ethernet connect systems at larger scales.
• Spectrum-X is NVIDIA’s Ethernet offering for AI data centers. Kagan says it combines endpoints and switches and brings features associated with InfiniBand to Ethernet. He claims it can nearly double data-center networking performance compared with conventional Ethernet.

For connections between facilities, the conversation emphasizes congestion control and traffic management to reduce jitter and make latency more predictable. It also acknowledges that the speed of light limits how quickly distant sites can communicate.

Why AI demands more computing

Kagan traces AI’s growth to programmable GPUs, which made it possible to use graphics hardware for general-purpose computing. He points to AlexNet’s success running on GPUs as a milestone that helped make neural networks more practical.

He contrasts the rapid growth in AI compute requirements with the slowing of Moore’s Law. As models and datasets grow, a single chip—or even a single rack—may not be enough. He says systems with around a thousand GPUs are a step toward much larger deployments, and that million-GPU systems are approaching, though he does not cite an official announcement.

NVIDIA is also building large, modular AI data centers to test whether such systems can be built and operated as proposed. Kagan says the company aims to preserve standard interfaces and make the architecture available to partners, who can adopt it as-is or modify parts of it.

Future applications: physical AI

The discussion describes physical AI as a major next step. Robots could be trained using simulations of the physical world, where they can gain experience safely and more frequently than in real-world settings. They would also need onboard computing for immediate responses, since connectivity can be unreliable or introduce latency. Higher-level systems and data centers could support other tasks.

Kagan suggests that robots could share experience gathered in the field, allowing models to be refined and updates distributed back to robots—a process he characterizes as federated learning.

Background and advice

Kagan recounts starting at Intel, co-founding Mellanox to build high-speed networking, and later working with NVIDIA before Mellanox was acquired. He argues that networking expertise developed for cloud and high-performance computing also became important for AI systems.

His advice to younger people is to stay curious, pursue genuinely new opportunities rather than simply chasing existing competitors, and measure improvement against one’s own past performance. He also says he would consider studying computational biology because it offers a way to understand complex biological systems.

Sources and content type

• Main speakers: David Bombal (host and interviewer) and Michael Kagan (NVIDIA CTO and Mellanox co-founder).
• Reviews, guides, or tutorials: None; this is an interview and industry discussion.