The Engineering Unlocks Behind DeepSeek | YC Decoded

Overview

DeepSeek (a Chinese AI lab) released R1, an open-source reasoning model fine-tuned from its V3 base model. R1 attracted major public attention because it claimed near-OpenAI-level reasoning at much lower cost and was freely available. The release triggered social, media, and market reactions (notably large Nvidia market-cap swings). Much of R1’s algorithmic foundation was described in earlier DeepSeek work (V2, V3, and a math paper); R1 is primarily V3 plus RL-focused training.

Model stack and timeline

1. Feb 2024 — Math paper: introduced techniques relevant for numeric reasoning.
2. May 2024 — V2: introduced several building blocks used later.
3. Dec 2024 — V3: general-purpose base model (comparable to GPT-4o / Gemini 1.5 / Claude 3.5) with many efficiency-focused innovations.
4. End of Jan 2025 — R1: a reasoning-specialized model fine-tuned from V3 using reinforcement learning; matched OpenAI’s 01 on some math/coding benchmarks.

Key technical innovations and product features

FP8 training and FP32 accumulation fix

• V3 is trained natively in 8-bit floating point (FP8) to save memory and cost.
• Periodic merges into an FP32 accumulator are used to avoid numerical drift and error accumulation.

Higher GPU utilization strategies

• Goal: increase model FLOPS utilization (typical FP8 MFU ≈ 35% is low).
• Software and algorithmic optimizations extract more compute from limited GPU clusters, important given export controls and hardware constraints.
• Highlights the advantage of integrated stacks (NVIDIA GPUs + InfiniBand + CUDA + libraries) versus piecemeal setups.

Mixture-of-Experts (MoE) architecture

• V3 has ~671B parameters but activates only ~37B parameters per token (sparse activation), yielding ~11× fewer active parameters versus dense large models and saving compute.
• DeepSeek introduced stabilizing techniques to make MoE training efficient and to increase GPU throughput.

Multi-Head Latent Attention (MLA)

• Compresses KV caches into a latent representation to drastically reduce KV storage (claimed ~93.3% reduction).
• Increases generation throughput (reported up to ~5.76×).

Multi-Token Prediction (MTP)

• Predicts multiple future tokens per step during training/inference, improving data efficiency and enabling speculative decoding to speed generation.
• Helps with representation planning and generation throughput.

Reinforcement learning for reasoning (pure RL + GRPO)

• R1 was trained with reinforcement learning focused on step-by-step reasoning rather than supervised chain-of-thought examples.
• Reward pipeline:
  • Datasets with verifiable outputs (math / coding).
  • Simple rule-based grading of final answers (accuracy / format).
  • Training algorithm: Group Relative Policy Optimization (GRPO), published Feb 2024.
• Emergent behaviors: extended chain-of-thought and backtracking (“aha” corrections) appeared from this RL pipeline.

Cold-start fine-tuning

• To prevent random bilingual switching in chain-of-thought outputs (English/Chinese mixing), DeepSeek first fine-tuned on structured reasoning examples before RL, improving readability and coherence.

Performance, accessibility, and cost notes

• Performance: R1 reached parity with OpenAI’s 01 on certain complex reasoning benchmarks. Shortly after R1’s release, OpenAI released 03 which outperformed both 01 and R1 on key benchmarks.
• Accessibility: DeepSeek published model weights and papers publicly; R1 can be downloaded, run locally, and customized — a major reason for the hype.
• Cost claims: the commonly cited $5.5M training cost refers only to the final run for V3 (not full R&D, earlier runs, or infrastructure costs). With the described techniques, other groups reproduced reasoning behaviors in smaller models for far less (e.g., a UC Berkeley demo estimated ≈$30 for a smaller instance).

Broader implications and takeaways

• Demonstrates that non-Western labs can compete by optimizing software, training recipes, and GPU utilization rather than only scaling hardware.
• Shows the value of open publishing (weights + papers) for accessibility and reproducibility.
• Points to further opportunities in rebuilding the stack: inference-layer tooling, GPU workload optimization, AI-generated kernels, and lower-cost intelligence for consumer/B2B apps.
• Short-term implication: rapid iteration cycles — new models from other labs can quickly overtake released methods.

R1’s release illustrated how software, training methodology, and openness can shift competitive dynamics even without extreme hardware scale.

Relevant papers / published sources

• DeepSeek V2 paper (May 2024)
• DeepSeek math paper (Feb 2024)
• DeepSeek V3 paper (Dec 2024)
• GRPO (Group Relative Policy Optimization) description (Feb 2024)
• R1 release materials (end of Jan 2025)
• UC Berkeley reproduction work applying R1 techniques to a smaller model

Main speakers / sources

• YC Decoded (video producer / narrator)
• DeepSeek (V2 / V3 / R1 papers and release materials)
• OpenAI (01 and follow-up 03 models referenced)
• NVIDIA (hardware, InfiniBand, CUDA context)
• UC Berkeley (reproduction demonstration)