It’s von Neumann’s fault | GPU · TPU · LPU

Overview

The video explains how GPUs, Google TPUs, and Groq LPUs approach the same challenge: keeping computation supplied with data quickly enough to run AI models efficiently.

Google TPUs

Why Google developed the TPU

Google researchers found that running neural-network inference on CPUs—or even then-current GPUs—would be too costly for a widely used speech-recognition feature. Because neural networks perform many predictable matrix multiplications, Google designed an application-specific integrated circuit (ASIC) for that workload.

TPU v1 and the memory bottleneck

The first TPU used a 256 × 256 grid of multiply-accumulate units (MACs), optimized for 8-bit integer operations. Its high peak compute depended on keeping those units supplied with weights and activations. However, its external DRAM bandwidth was far below what the MAC array could theoretically consume, potentially leaving the chip underused.

This illustrates the von Neumann bottleneck: data must travel between memory and processing units, and that movement can limit performance.

Systolic arrays

Drawing on computer scientist H. T. Kung’s work, the TPU moves data through a grid of processing elements in a regular, timed pattern. Weights can remain in place while activations flow through the array, allowing values to be reused and reducing memory traffic. Google’s TPU design applies this approach to matrix multiplication.

TPUs and GPUs

Google’s first TPU was deliberately specialized, with little control logic and a small instruction set compared with a general-purpose processor. The video says it substantially outperformed the Nvidia K80 on tested inference workloads while using less power.

Later TPUs and Nvidia GPUs both adopted high-bandwidth memory (HBM), improving bandwidth. Nvidia’s Tensor Cores perform many smaller matrix operations, giving a runtime planner more flexibility than a large, fixed matrix unit. The video notes that memory-bound workloads remained an issue for both.

Groq’s LPU approach

Groq’s language processing unit (LPU) uses on-chip SRAM for model weights rather than external DRAM. SRAM is faster and has more predictable access times, but it is much less dense, so each chip holds relatively little memory. The video gives the example of distributing a 70-billion-parameter model across hundreds of LPUs.

Static scheduling and predictability

Because SRAM access and operations have predictable timing, Groq’s compiler can schedule instructions, memory accesses, and chip-to-chip communication with clock-cycle precision. This can make execution latency consistent and predictable.

The trade-off is that compilation is more demanding and can be closely tied to a model’s shape and hardware layout. The video also describes how the system can route work to different experts in mixture-of-experts models while preserving a scheduled execution pattern.

Where each architecture fits

GPUs can process large batches efficiently, spreading the cost of moving weights from HBM across many tokens. LPUs can reduce delays during token-by-token generation, but their SRAM-heavy design can be expensive. The video presents the approaches as complementary rather than mutually exclusive.

A possible hybrid

LLM inference has a parallel prefill stage and a sequential decoding stage. The video suggests that GPUs suit prefill and attention, while LPUs can handle some of the large fully connected layers during decoding.

The growing KV cache is a constraint for LPUs because it can exceed their available SRAM. The video also describes a proposed system pairing an LPU rack with Nvidia GPU racks.

Nvidia’s positioning

Nvidia is presented as emphasizing GPUs’ broader role in accelerated computing—not only AI, but also scientific and data-processing workloads. The video closes by noting the growth of other AI-chip companies, while saying that some are less forthcoming about their technology.

Review and guide format

This is a technical explainer, not a product review or step-by-step setup guide. Its main tutorial-style section illustrates how a systolic array reuses data during matrix multiplication.

Main speakers and sources

• Julia Turc — narrator and presenter.
• Jonathan Ross — interviewee who discusses TPU development and Groq’s LPU and compiler approach.
• Abdel Sghiouar — Google Cloud developer advocate, identified in the video’s context as an interview source.
• Other sources discussed: John Backus’s explanation of the von Neumann bottleneck; H. T. Kung’s work on systolic arrays; Google’s TPU design and papers; and an unnamed Nvidia representative discussing accelerated computing.