Build DeepSeek from Scratch: Series Introduction

Summary (tech-focused)

The speaker introduces a planned YouTube series, “Build DeepSeek from Scratch”, arguing that most people are missing the real value in the “DeepSeek wave.” Instead of focusing on flashy applications or using existing APIs, the goal is to understand and rebuild DeepSeek’s core architecture and training pipeline from first principles, including detailed coding and mathematical components (e.g., matrix multiplication).

Main claims / context about DeepSeek

DeepSeek’s rise is presented as both technical and geopolitical/economic, referencing reports that it:

• Was perceived as matching or exceeding top OpenAI models
• Was much cheaper to build and cheaper to run
• Was open source / permissively licensed, which increased interest and sparked policy discussion in other countries

Examples mentioned:

• Companies reportedly shifting from OpenAI API calls to DeepSeek APIs
• Countries debating building foundational models (e.g., India)

Speaker’s motivation and learning gap

The speaker says they couldn’t find tutorials or videos that truly explain the “nuts and bolts”, specifically:

• Code from scratch
• Mathematical details

Existing materials are described as:

• Too “flashy”
• Too short/high-level
• Focused on building applications on top of existing DeepSeek components (e.g., guides assuming DeepSeek R1 already exists)

What the series will cover (planned curriculum)

The intro outlines a structured plan in two major parts.

1) Architecture chapters (build modules from scratch)

Key architecture elements explicitly mentioned:

• Mixture of Experts (MoE)
• Multi-head latent attention
• Quantization
• Multi-token prediction

The speaker emphasizes that each topic is non-trivial and will be taught deeply, including:

• Extensive written notes (they mention preparing ~50 pages for multi-head latent attention)
• Implementations in code
• Comparisons to traditional approaches (e.g., latent attention vs standard multi-head attention)

2) Modeling chapters (training / RL pipeline)

Key training methods explicitly mentioned:

• DPO (Direct Preference Optimization)
• Po (as referenced in subtitles; likely another preference/RL variant)
• GRPO (Group Relative Policy Optimization)

The goal is to:

• Teach reinforcement learning components step-by-step
• Explain how these methods combine into the broader training pipeline

End goal / replication target

The series will culminate in recreating DeepSeek R1-like capabilities, including:

• Rebuilding the model components
• Running it on GPUs

Estimated workload:

• ~20–25 hours, possibly more

Teaching approach / deliverables

• A video series plus accompanying notes/book
• The speaker is also working on a book based on their notes, with planned later publication
• Strong emphasis on illustrative/visual explanations
  • The speaker believes visuals and code are more effective than text-only blog explanations

Main speakers/sources

• Speaker: Dr. Raj Dandera (co-founder of Viuma; PhD in Machine Learning, MIT; earlier background referenced as IIT Madras)

Referenced sources (context only, not direct instructional sources):

• DeepSeek researchers/engineers and DeepSeek papers/releases (from early 2024 work to DeepSeek R1 in Jan 2025)
• Mentions of OpenAI models/APIs and related market/news commentary
• Mentions of online content such as tutorials/guides (described as insufficient for “from scratch” learning)