NVIDIA RTX Spark Chip Architecture is INSANE

Key technological idea / problem being addressed

• Traditional PC architecture places the CPU and GPU on opposite sides with separate system memory and VRAM, connected via a bus (e.g., PCIe).
• This separation becomes inefficient for large language model (LLM) workloads.
• Example: a 70B-parameter model can require roughly ~70GB just to load, while a typical GPU may have only ~32GB VRAM.

Workarounds today

• Cloud inference billed per token.
• Multi-GPU setups (e.g., several GPUs to “stitch” memory together), typically at the cost of high power draw.

---

What Nvidia RTX Spark is (product/architecture concept)

Unified memory on the same chip

• RTX Spark integrates CPU and GPU into a single SoC with a shared memory pool.
• This targets the inefficiency of copying data across PCIe between CPU memory and GPU VRAM.

Platform and collaboration details

• Compared with Apple’s approach:
  • Apple has offered unified memory since ~2020, but macOS-only.
  • RTX Spark targets Windows laptops later this year.
• Collaboration:
  • Nvidia designed the compute/GPU side.
  • MediaTek handled system integration, including the memory controller and LPDDR5X interface.

---

Hardware configuration described

Single package with two main blocks

• CPU side
  • 20-core ARM
    • 10 high-performance Cortex X925
    • 10 efficiency A725
• GPU side
  • Blackwell architecture
    • 48 streaming multiprocessors
    • 6,144 CUDA cores

Unified memory

• Shared pool of up to 128GB LPDDR5X
• Eliminates the usual “copy step” across PCIe, enabling the GPU to read directly from the same memory used by the CPU.

---

Internal interconnect

• Internal chip-to-chip link: NVLink C2C
• Coverage initially cited 600 GB/s, later clarified as die-to-die interconnect bandwidth, not DRAM bandwidth.

---

Performance / benchmark highlights (and what was corrected)

Benchmark expectations and comparisons

• Estimated 3DMark Time Spy (core-based) score: ~21,000
• Snapdragon X2 Elite score in the same test: ~4,200

Developer workload: Clang compilation

• The 20-core ARM chip:
  • beat base M5 by ~54%
  • but was about ~7% behind M5 Pro

Bandwidth correction

• Early reporting mistakenly treated 600 GB/s as memory bandwidth.
• Corrected LPDDR5X throughput cited: ~273 GB/s (described as a key “wall” for AI inference throughput).

---

AI inference implications (token-per-second reasoning)

Estimating memory reads per token (FP)

• For FP inference, a 70B model in FP format implies roughly ~70GB of weights read per token.

Bandwidth-based token ceiling

• With ~273 GB/s memory bandwidth:
  • estimated ceiling is roughly ~4 tokens/second.

Mitigations mentioned

• Use FP4, supported natively by Blackwell tensor cores.
• Use mixture-of-experts (MoE) so only a fraction of weights need to load per token.

---

CPU / GPU ecosystem compatibility and developer relevance

CUDA on Windows ARM (key milestone)

• Claimed: CUDA now runs on Windows ARM laptops (previously not available).
• Why it matters:
  • Much AI developer tooling is tied to Nvidia/CUDA.
  • Python scripts and TensorRT models carry over more easily.
  • Docker may require ARM64 images, but CUDA underneath stays consistent.

Windows ARM GPU acceleration availability

• Apps cited as receiving full Blackwell GPU acceleration:
  • Photoshop
  • Premiere
  • DaVinci Resolve
  • Blender
• Reported access includes:
  • CUDA
  • Tensor Cores
  • hardware ray tracing

Handling legacy x86 apps

• Microsoft Prism emulator translates, while graphics calls go directly through the GPU driver.

---

Competitive positioning / market analysis

Nvidia RTX Spark is framed as competing against:

• Apple silicon unified-memory laptops (no longer the only mainstream unified-memory + GPU option)
• x86 modular PC architecture (CPU/GPU/memory separated), dominant for ~40 years
• The possibility of Qualcomm being pushed out of the premium creator graphics tier
• Intel/AMD modular SOC designs, which may be harder to justify versus unified designs at lower power

Power / cost trade-off context

• Apple price references:
  • Lower-tier M1 starting around $1,800
  • M1 Max (128GB) estimated around $2,900

---

Main speaker(s) / sources mentioned (end)

• Nvidia (RTX Spark architecture/compute design)
• MediaTek (system integration + memory controller)
• Apple (unified memory approach since ~2020)
• ARM (Cortex X925 / A725 CPU cores)
• Blackwell (GPU architecture)
• Benchmark sources / tools:
  • 3DMark Time Spy
  • Geekbench
  • clang compilation benchmark