Oxide and Friends 2/3/2025 -- AI Disruption: DeepSeek and Cerebras

Tech-focused summary of the subtitles (Oxide and Friends, 2/3/2025)

1) Why DeepSeek’s R1/Distilled models caused major industry buzz

DeepSeek’s release of “R1”-style reasoning capability (a reasoning model family) drew unusually large attention across Silicon Valley and markets.

Key contributing factors discussed:

• Lower training cost claims (e.g., a cited “$6M training cost” rumor) that made the story both technical and financial.
• Architectural efficiency: movement toward Mixture-of-Experts (MoE) / sparse expert routing, improving compute efficiency for a given input/output.
• Reasoning at inference time (“thinking longer”): performance improves not only by scaling model size, but by spending extra compute during inference before answering.

The discussion contrasted DeepSeek with earlier reasoning approaches (e.g., “o-series” style reasoning) that allegedly didn’t generate the same level of buzz—suggesting DeepSeek combined efficiency + narrative + results in an unusually attention-grabbing way.

2) Cerebras demonstration: DeepSeek run on Cerebras hardware is “instant”

Speakers described running DeepSeek on Cerebras with very high throughput / low latency:

• An anecdote claims answers returned “instantaneously”, giving the impression the system was “broken” due to far faster responses than typical GPU/chat experiences.

The specific model described:

• A “70B Llama model” distilled variant of DeepSeek hosted on Cerebras inference.
• DeepSeek R1 was characterized as a more complex “671B MoE” model, but Cerebras wasn’t initially running the full MoE directly.
• Instead, they used a distilled 70B model that is described as bitwise compatible/aligned with what Cerebras already hosts, then swapped weights and optimized.

3) Cerebras performance claims and why it’s fast (architecture explanation)

Cerebras frames its speed via a fundamentally different memory and interconnect design.

Hardware platform: Wafer Scale Engine

• Wafer-scale chip (“wafer scale engine”):
  • About 8.5 × 8.5 inches of unbroken silicon (dinner-plate scale) on a roughly 300mm wafer manufacturing basis.
  • Approximately 900,000 cores.
  • About ~125 petaflops sparse FP16 compute (per chip).
  • Roughly ~44GB of on-chip SRAM (for inference).

Key reason for speed: weights in on-chip SRAM

• Cerebras argues that keeping model weights local avoids a common GPU bottleneck:
  • GPUs often bottleneck on off-chip HBM memory bandwidth, because weights must be evicted/reloaded repeatedly during token generation.
• Cerebras claims it avoids that penalty by placing weights in SRAM so inference repeatedly streams through local weights.

Inference mechanics (as described)

• For a 70B parameter model:
  • Traditional GPUs must move/copy weights in batches across the memory interface.
  • For subsequent tokens, the system must effectively re-read/reload weights again, limiting performance.
• Cerebras claims weights “hug” the compute:
  • Load weights once into SRAM, then generate tokens by repeatedly computing through parameters without the same external-memory traffic penalty.

Scaling to larger models

• For very large models (e.g., 70B ≈ ~70GB of parameters in the framing given), the model may not fit into a single wafer’s 44GB SRAM, so Cerebras chains multiple wafer-scale systems.
• They argue chaining doesn’t cause the same slowdown as GPUs because activations moving between layers require less bandwidth than reloading full weights repeatedly.

Benchmark-style claim

Cerebras cites published analysis (e.g., “artificial analysis”) suggesting:

• GPU inference providers often hit a performance ceiling due to memory bandwidth limits.
• Cerebras has a different bottleneck profile (“compute-bottlenecked”), yielding large speedups.

A specific number claimed:

• For a DeepSeek R1 distill Llama 70B variant: ~57× faster than GPU instances
  • With the caveat that you can’t simply scale GPUs linearly to match.

4) Hardware–software co-design: models adapted to run efficiently on the backend

A recurring theme is hardware/software co-design:

• DeepSeek’s design choices (e.g., MoE/sparsity) are discussed as aligning with how the backend is intended to operate.

Cerebras’ motivation is framed around AI workload traits:

• AI needs lots of FLOPs, but can use low precision (8/16/32-bit rather than 64-bit).
• Sparsity (MoE-like structures).
• Data movement locality: inference/training benefit when data/activations are near compute with massive memory bandwidth.

5) Software ecosystem: “Meet users where they are” (not CUDA-first)

Cerebras emphasizes it does not rely on CUDA at the lowest level:

• Its chip uses a different instruction set and microcode.
• A compiler sits on top of that.

Yet the platform integrates upward into common tooling:

• PyTorch integration for training.
• Inference aligns with typical deployment patterns (e.g., apps calling an inference endpoint).

The stated goal: let developers use standard model descriptions/workflows, avoiding the need for deep low-level GPU kernel programming.

6) Open weights impact: why open models accelerated adoption and innovation

The discussion argues open-weight models (e.g., Llama, DeepSeek) reshape the ecosystem:

• Hardware vendors can demonstrate advantages because users can run models on their own stack.
• Developers can fine-tune and build on top, increasing experimentation speed.

Open models also address enterprise concerns:

• Data privacy / sovereignty: enterprises worry proprietary models might train on their data.
• Open weights enable on-prem or controlled deployments.

Closed vs open:

• Closed models can still win on performance/accuracy in some specialized scenarios.
• But open models drive accessibility, customization, and developer momentum—making it harder for proprietary advantage to persist based only on “model access.”

7) Reasoning-model behavior and “chain-of-thought” exposure

A notable behavior comparison:

• DeepSeek R1 is described as showing more visible chain-of-thought than OpenAI-style models (described as more circumspect).

Reactions mentioned:

• Some describe the visible internal reasoning as wild/chaotic/self-doubting (“manic” vs “reassuring”).
• There were cases of hallucinated API/function names that needed correction.
• The chain-of-thought visibility is also framed as educational—revealing where the model’s limitations show up.

8) Broader implications: the “dial-up to broadband” moment for AI applications

The conversation uses infrastructure metaphors to argue that better performance changes what becomes feasible.

Faster inference is framed as enabling new categories of AI applications:

• multi-agent workflows
• multimodal interaction
• fast voice + reasoning
• real-time agentic operations

They compare it to networking’s shift:

• early AI = “dial-up era” (limited testing/use beyond simple chat)
• improved “bandwidth” = unlocking whole industries

---

Mentioned sources / main speakers

• Adam (Cerebras): hardware/software discussion, performance + architecture explanations
• James (Cerebras): running distilled DeepSeek Llama 70B; model compatibility/weight swapping
• Andy (participant): context/analogy; discussion of open models + ecosystem implications
• Brian (host/interviewer)
• Simon Willison / Simon Willson: referenced as a prior guest; tied to DeepSeek predictions
• TJ Watson Jr at IBM: historical memo about CDC/IBM competition; quoted
• GPU provider references: OpenAI, NVIDIA, NVLink, H100, H800, GTC, plus “artificial analysis” for benchmark-style observations
• Eric Meyer: referenced regarding efficiency and training/compute critique