High Bandwidth Flash Is Coming

Main technological concepts (LLM inference + memory bottlenecks)

Prefill vs. Decode

• Prefill: Processes the input in parallel (e.g., prompt, system prompt, prior replies, tool calls, multimodal data).
  • Transformer attention produces Query/Key/Value, where Key/Value are stored.
• Decode: Generates output one token at a time, consulting the KV Cache for prior context.

KV Cache purpose

• Stores attention Key/Value from earlier tokens so the model doesn’t repeatedly recompute attention over the entire prompt/state.
• Performance trade:
  • Prefill is more compute-bound (parallelizable).
  • Decode is more memory-bound (waiting on data loads), making memory bandwidth/latency critical for token throughput.

---

Why memory tech matters now (“cultivating MASS”)

Two compounding growth drivers

1. LLM weights (e.g., multi-trillion-parameter models)
   • Example mentioned: Kimi K3 at ~2.8T params; terabyte-scale at 8-bit.
2. KV Cache size
   • Grows with long-context and long-running agents (up to ~1M tokens mentioned).

Current mitigation idea: KV compression/blocking

• Splits KV into blocks, moving less-frequently used parts to SSD while keeping “hot” parts in HBM.
• Downside: compression can reduce detail fidelity (likened to a “blurry JPG”).

---

HBM: benefits and emerging limitations

HBM overview

• 3D-stacked memory with a logic base die and stacked DRAM dies via TSVs (through-silicon vias).

Core criticism / concern

• As stacks get taller to add capacity, bandwidth per layer/die can get diluted due to TSV/stack trade-offs.
• Potential economics/performance issue:
  • Increasing capacity may become more expensive and may reduce effective per-core throughput at extreme stack heights.

Reported industry sentiment

• Discussion referenced Irrational Analysis suggesting HBM could be a “mistake.”
• During Hot Chips 2026, a SemiAnalysis/analyst-style question highlighted bandwidth dilution when using taller stacks (more layers) instead of improving speed.

---

High Bandwidth Flash (HBF): what it is and why it’s being considered

What HBF claims to do

• Concept: stack/connect flash (NAND) dies similar to how HBM stacks DRAM dies, using advanced packaging (not monolithic 3D NAND).
• Capacity advantage:
  • Described as potentially 8–16× higher per stack than HBM (example given: ~512GB vs ~36GB per HBM4 stack).
• Bandwidth goal:
  • Keeps “high bandwidth” semantics and may increase bandwidth by exposing more NAND cells to the GPU.
  • Effective figure mentioned: up to ~3 TB/s (with comparisons to HBM3E/HBM4 numbers).

Key downsides / constraints (“the catch”)

Because NAND isn’t DRAM, the discussion emphasizes three major limitations:

1. Much slower program/write

   • Program/write is ~2 orders of magnitude slower than HBM.
   • Leads to latency concerns (programming is slower than reading).

2. Endurance / wear-out

   • NAND cells have limited write cycles (framed as thousands of cycles in an order-of-magnitude comparison).
   • Implication: workloads should be read-mostly, with careful write minimization.

3. Power and thermal costs

   • Large HBF “cubes” consume more power and generate more heat.
   • Heat can cause throttling, worsening effective endurance/performance.

---

Product/support ecosystem timeline and standardization

Earlier research lineage

• 2019 IEEE 3D Systems Integrated Conference work introduced “High Bandwidth NAND” using TSV/bump-less approaches.

Commercial/industry milestones (as described)

• Feb 2025: SanDisk introduces HBF at its Future FWD investor day.
• May 2024: Patent filing mentioned.
• July 2025: HBF Technical Advisory Board formed
  • Includes David Patterson and Raja Koduri.
• Aug 2025: MOU with SK hynix for spec standardization.
• (Timeline described): Google and Tenstorrent join a consortium.
• ~Aug 2026: Open Compute Project public spec release
  • Version 0.7.0.
• Samsung has a related approach (zNAND-O), described as potentially similar but not guaranteed to match HBF.

---

“Where does it go?” (integration strategy for AI systems)

Proposed place in the memory hierarchy

• The pitch: place HBF alongside HBM at Tier 1 (near the GPU).
• Use HBM for “hot” data and HBF for “warmer” / larger data.
• Constraint: GPU interposer footprint (capacity trade-offs must fit physically).

Example scaling math (as stated)

• A GPU with 8×24GB HBM (~192GB total) could become 6 HBF + 2 HBM, reaching ~3.12TB total memory.

Major systems caveat: token throughput isn’t automatically cheaper

• Hot Chips 2026 systems point:
  • Even if capacity is cheap, token generation speed depends on whether the system can supply data fast enough from HBF.
• Requires extensive software support and intelligent data placement.

---

Software/workload implications emphasized in the video

“HBF Sucks?” paper (described)

• Argues:
  • HBF is not a drop-in replacement for SSDs.
  • Software must carefully decide what goes into HBF—example: shared prompts.
• Model-structure fit:
  • A Mixture-of-Experts (MoE) model may be a better candidate because only a subset of experts is read per token, making lower bandwidth per GB potentially more acceptable.
• Contrasts with a naive assumption that “more capacity = faster performance.”

---

Historical analogy: Intel Optane lessons (failure modes to avoid)

Comparison to Optane (3D XPoint / PCM)

• HBF is compared to Optane, which was marketed as “in-between” NAND and DRAM (“warm data” with DRAM-like latency).
• Optane struggled due to market/price pressure and strategy issues, despite promising memory technology.

Takeaway for HBF

• Avoid repeating Optane’s pitfalls in:
  • controller/productization,
  • go-to-market strategy,
  • and ensuring the platform actually delivers promised performance with software support.

---

Reviews/guides/tutorial references

• No direct step-by-step how-to guide is presented.
• The discussion cites technical papers/specs as evidence, including:
  • Transformer inference concepts (prefill/decode/KV cache)
  • 2019 IEEE “High Bandwidth NAND”
  • 2026 Hot Chips system integration discussion
  • “HBF Sucks?” performance/placement analysis
  • Open Compute Project spec v0.7.0

---

Key speakers/sources mentioned (by the video)

• Irrational Analysis
• SemiAnalysis analyst (questioner at Hot Chips 2026, quoted via Irrational Analysis)
• Micron, Samsung, SK hynix (memory makers presenting at Hot Chips 2026)
• A. Agrawal and R. Giduthuri (Hot Chips 2026 architects discussing HBF integration)
• Koji Sakui and Takayuki Ohba (authors of the 2019 “High Bandwidth NAND” concept)
• SanDisk
• David Patterson and Raja Koduri (HBF Technical Advisory Board)
• Google, Tenstorrent (joined the consortium)
• Open Compute Project (HBF spec v0.7.0 release)
• Intel Optane history: Intel and Micron (Optane/3D XPoint background) and Alper Ilkbahar (linked to HBF leadership and previously Optane group leadership)
• Samsung (mentioned for zNAND-O alternative)
• Hot Chips / Hot Chips 2026 (talks and Q&A)