Local AI Is Now 6x Faster on Your PC

Key technological idea: local “expert caching” for MoE models

The video explains why the local AI engine Strata can run Qwen 3.8 FlashNext much faster than llama.cpp on consumer PCs.

The core breakthrough: “expert caching” for Mixture-of-Experts (MoE)

In an MoE model, each token activates only ~2% of the experts—for example:

• Each token uses 10 of 512 experts per layer
• Across a full forward pass, a token touches only a small subset overall

That means you don’t need all expert weights in VRAM at once. Strata uses a two-tier approach:

• Frequently used experts stay on the GPU (“counter”)
• Less-used experts remain in system RAM (“pantry”)

---

Reported performance: “6x faster,” with a focus on reading vs writing

The headline claim compares Strata to llama.cpp, but the video emphasizes that the biggest gains come from prompt processing (“reading”), not just generation (“writing”).

Example configurations mentioned

Developer PC

• Ryzen 5 7600, 64GB DDR5, Windows, RTX 5070 (12GB)
• Same model/version:
  • llama.cpp: ~15 tokens/sec writing (reads slower)
  • Strata: ~65 tokens/sec writing (and much faster reading)

Laptop user

• RTX 5070 Ti (12GB), 64GB RAM
• Reported:
  • llama.cpp: ~23 tokens/sec writing
  • Strata: ~51 tokens/sec writing
• Reading improvements were far larger:
  • llama.cpp read: ~100 tokens/sec
  • Strata read: ~1500 tokens/sec for large prompt contexts

What the “6x” depends on

• Time-to-first-word (prompt → first output) drops drastically with Strata due to faster prompt reading.
• The “6x” figure is described as derived from write-speed comparisons in some setups.
• The more “wow” factor is reading speed:
  • Reported ~14.5x–17.5x gains on similar rigs
  • Sometimes ~15x+, depending on model/version pairing

---

How Strata achieves speed (three main tricks)

1) Expert caching strategy

• Fill limited VRAM with experts likely to be needed soon.
• As the conversation changes, update which experts are “hot” and should remain on GPU.
• This avoids llama.cpp’s bottleneck where many experts must be fetched across the slower VRAM↔RAM path (described as the “hallway” problem).

2) CPU/GPU overlap

• While the GPU processes current layers, the CPU precomputes/carries forward next experts.
• The video frames it as balancing:
  • CPU waiting on GPU
  • GPU waiting on CPU
  • with roughly matched time slices to reduce stalls.

3) Multi-Token Prediction (MTP) draft/verify

• Strata uses a drafting layer that proposes up to 3 tokens ahead.
• The full model then verifies those tokens (up to 48 layers).
• Reported generation improvements: ~1.6–1.8x faster
  • Example token/sec shift: ~47–57 → ~82–92
  • (with similar output quality claimed)

---

Prompt length and “reading” pipeline improvements

• Strata processes long prompts in chunks up to 8,192 tokens.
• It overlaps processing so that while the GPU works on one part:
  • experts for the next part can be moved/prepared via PCIe

A cited example:

• Time-to-first-word improves from ~15 seconds to ~12 seconds for ~32k context on an RTX 5070.

The read-speed table reportedly supports larger theoretical gains on longer prompts.

---

Compatibility requirements and practical install notes

RAM vs VRAM emphasis

The video claims this approach is more RAM-dependent than VRAM:

• 64GB RAM: all versions work; recommended quantization is IQ2XS
• 48GB RAM: only certain quantizations fit (e.g., Q20 / IQ2XS)
• 32GB RAM: a later update added a programming-oriented variant (keeping ~half the experts)

VRAM requirement

• NVIDIA RTX 20–50 series with 12GB+
• AMD options exist but status is described as OS-dependent

Supported platforms

• Windows/Linux listed
• macOS not listed (though a user reported an Apple Silicon-specific engine outside the main package)

Disk and first run behavior

• Needs about 80GB free
• Installer may “freeze” for 1–3 minutes (startup work such as filling the “pantry”)

Multi-user / concurrency behavior

• Not a server-style multi-user setup by default:
  • handles one request at a time
  • typically maintains one conversation
  • optional caching may change behavior

---

Model, versions, and licensing context

Model

• Qwen 3.8 FlashNext (architecture terms: 125B parameters)
• Full precision is very large (~354GB), so quantized variants are required in practice.

Quantization examples

• Q20: ~66.4GB (too large for VRAM; depends on caching/loading strategy)

Licensing and project activity

• Strata is described as free
• MIT licensing is mentioned (since Sept 28 per the video)
• Repository activity: rapid iteration with multiple releases between Sept 24 and Sept 27, reaching ~v0.39 by Oct 3; also noted as having thousands of stars

---

Accuracy / output-matching concerns (review-style scrutiny)

The video stresses that speed claims should be checked against output fidelity.

Main question: exact output match

• Does Strata produce exactly the same output as a normal engine?
• Or do speed shortcuts alter tokens?

What was reportedly checked

• Developer responses included limited testing:
  • some tests claimed identical results (e.g., Blender + JavaScript)
• Community comparisons:
  • one user reported 99.4% similarity with llama.cpp on a short phrase/code section
  • divergence appeared later in a Python function

Important caveat: expert cache might affect text

• A quoted message suggests that with expert cache enabled, output might differ from runs without it:
  • timings may be real
  • but text may vary

Additional version caution

• Early versions allegedly ignored temperature adjustment
• The video mentions many open issues at the time

---

“Tutorial/review” angle: what to choose and when it’s worth it

• If you already have enough RAM, try Strata now (described as free).
• If you must buy hardware:
  • consider whether cloud inference might be cheaper than building a RAM-heavy local setup

Cloud cost framing (example)

• Compares local RAM costs vs Qwen cloud pricing per token (entry/exit),
• suggesting local can be cost-effective if you already have/plan sufficient RAM.

Mentioned variants / experimental features

• Swift 1.5 variant: claims fewer “thought tokens”
• Experimental velocity projection:
  • described as potentially increasing perplexity
  • disabled by default

---

Main speakers / sources (as referenced)

• “He” / the developer posting on Reddit A Strata developer explaining expert caching and replying on r/localllama

• Reddit users on r/localllama Measurement reports and concerns about output matching

• Strata GitHub repository README, documentation, release notes, version tables (primary technical reference)

• Llama.cpp Used as a baseline engine and referenced as a target for similar optimizations (including mention of a PR to add expert caching)