Video summary

Local AI On Apple Silicon uses 7X Less RAM

Main summary

Key takeaways

Technology

Main claim / result

  • A local-run 26B parameter Mixture-of-Experts (MoE) model can run on Apple Silicon with ~2 GB RAM.
  • The author reports ~23 tokens/second on an M3 Max.
  • The project is framed as “usable” despite being extremely RAM-efficient.

How it avoids high RAM needs (vs. Gemma 4)

  • Gemma 4 is an MoE model with:
    • 128 expert feed-forward blocks
    • a small router
  • For each token:
    • the router selects the top 8 experts
    • so only about ~3.9B parameters are active per token
    • even though total parameters are ~26B
  • The core memory idea:
    • keep only the parts needed at all times in RAM
    • stream expert weights from SSD on demand

Key performance idea: “Turbo Field Fair”

Experts are not fully loaded into RAM. Instead, the model is split into two “piles”:

  1. Resident pile (~1.35 GB) Components needed every token, including:

    • attention
    • router
    • embeddings
    • one shared expert (always runs) These are memory-mapped from disk and kept resident while the model is loaded.

  2. Non-resident experts pile (~12.9 GB on SSD)

    • 30 layers, each with 128 experts
    • expert weights are stored on SSD
    • pulled into memory in small chunks (a few MB) only when required

Token generation flow (why streaming is hard)

  • Each token is processed through 30 layers sequentially.
  • Each layer performs:
    1. Attention (uses the resident ~1.35 GB; no disk access)
    2. Router chooses which experts (from 128) are needed, typically top 8
  • Why prefetching is difficult:
    • expert choice depends on computations up to that point in the token
    • therefore the system may need to fetch expert weights just-in-time
    • uncached fetches can cause CPU stalls, potentially up to 30 times per token

Why it’s “specifically a Mac project” (Apple Silicon vs PC GPU)

  • On PCs with discrete GPUs:
    • weights often move SSD → system RAM → GPU VRAM
    • this can require multiple copies and PCI transfers, hurting performance
  • On Apple Silicon (unified memory):
    • CPU and GPU access the same physical memory
    • using Metal buffers allows the CPU to read bytes off SSD into memory that the GPU can use directly

Disk/weight format optimization

  • Turbo Field Fair uses a file layout matched to the Metal GPU kernels.
  • Weights are stored in the format needed including 4-bit quantized values.
  • The goal is to avoid an intermediate unpack/re-encode step before GPU execution.

Asynchronous pipeline / “fetch overlap”

  • While the CPU fetches expert weights:
    • the GPU continues working on:
      • the shared expert
      • and other resident components
  • Multiple work streams run asynchronously to hide latency and reduce effective waiting time.

Caching strategy to reduce SSD reads

  • Per layer:
    • 16 of the 128 experts are kept parked in memory.
  • If the router selects a cached expert:
    • it runs immediately
  • Otherwise:
    • the required expert is loaded from SSD
    • the system evicts using an LFU cache policy:
      • evict least frequently used rather than LRU
  • Rationale:
    • routing is expected to be predictable (some experts used often)
    • if routing were random, cache hit rates would drop and performance would suffer

Practical guide/tutorial elements

  • The project includes instructions such as:
    • running commands from the GitHub repo to clone the project
    • launching it as a Mac app
  • After installing:
    • download the model
    • load the model
    • send the first message to test generation speed and measure memory footprint

Main speakers / sources

  • Primary speaker: the video’s narrator/host (creator), explaining “Turbo Field Fair”, including how it compares to Gemma 4.
  • Project source: the Turbo Field Fair GitHub repository (where clone/run commands are provided).

Original video