Running a 35B AI Model on 6GB VRAM, FAST (llama.cpp Guide)

Can you run a 35B MoE model on a 6GB GPU? (Yes—using llama.cpp + targeted flags)

The video demonstrates running Qwen 3.6 35B A3B (a mixture-of-experts model where only ~3B active parameters per token; 8 experts activated per token) on an 8-year-old GTX 1060 (6GB VRAM) with an i3-8100 CPU. The goal is not just “it runs,” but usable real-time speed and large context.

---

Baseline: naive GPU/CPU layer split (too slow)

• Approach: split the model with something like ~20 GPU layers and the rest on CPU/RAM (e.g., dash NGL / GPU layer count style).
• Result: about 3 tokens/sec (“satellite phone territory”).
• Why it’s slow (MoE-specific bottleneck): moving a layer to CPU moves its whole expert blocks, causing per-token PCIe transfers (CPU↔GPU), which chokes the bus.

---

Key optimization #1: pin MoE experts to CPU (--n-cpu-moe 41)

• Flag: --n-cpu-moe 41
• Idea: keep fast, small parts on GPU, but pin the MoE expert blocks to CPU.
• Why it helps: GPU computes, then requests needed experts; since most experts are “sleeping” most of the time, GPU VRAM isn’t wasted.
• Result: ~3 tok/s → ~10 tok/s (about +230%).

---

Key optimization #2: disable mmap for predictable loading (--no-mmap)

• Flag: --no-mmap
• Reasoning: llama.cpp may rely on OS paging (mmap-like behavior), which can trigger disk reads/page faults when an expert isn’t resident.
• With --no-mmap: load the full model into RAM upfront to avoid mid-inference disk stalls.
• Result: ~10 tok/s → ~13.5 tok/s (about +35%).

---

Key optimization #3: move more layers/experts onto GPU (trade VRAM for speed)

• Change: decrease CPU MoE setting from 41 to 35
• Effect: more MoE expert blocks pulled back to GPU, reducing PCIe traffic.
• Reported VRAM usage: ~4GB → ~5.5GB
• Result: ~13.5 tok/s → ~17 tok/s
• Trade-off: larger GPU footprint reduces maximum KV cache / context window.
  • Context drops from 100k → ~64k tokens initially.

---

Key optimization #4–5: reclaim huge context with Turbo Quant (KV cache compression)

Context length consumes VRAM linearly due to the KV cache (stores keys and values per token per layer). Even with Q8 cache, more compression is needed to increase context.

The video references Google DeepMind’s Turbo Quant:

• Keys: 4-bit quantization (“turbo four”)
• Values: 3-bit quantization (“turbo three”)
• Quality claim: in this Q3/Q4 region, answers are close to Q8.

Flags (as described)

• “turbo four for the keys”
• “turbo three for the values” (Exact CLI flag names weren’t shown in the subtitles, but correspond to Turbo Quant key/value modes.)

Context experiments on 6GB VRAM (keeping speed high)

Starting point after speed tuning:

• ~17 tok/s, context ~64k

Steps:

• 128k: fits
  • reported VRAM ~5.3GB
• 256k: initially OOM
  • fix: move one more MoE setting --n-cpu-moe 35 → 36
• Final successful run:
  • 256,000-token context
  • still ~17 tokens/sec
  • reported VRAM ~5.9GB / 6GB

Practical payoff: supports “small book” paste-in and codebase-scale Q&A without losing early pages due to context limits.

---

Production stability optimization: mlock + Docker permissions

Problem described:

• After long runs, memory can get paged out, causing page faults and stutter / random slow tokens.

Fix requires 3 places:

1. LXC container permission to lock memory
2. Docker capability: IPC lock (cap_ipc_lock)
3. llama.cpp mlock flag

Result:

• mlocked 16 GB (reported via meminfo)
• Multi-day slowdown disappears
• Speed remains ~17 tok/s (fresh boot helps because experts are already cached)

---

Speculative decoding attempt: doesn’t work for this MoE + SSM model

The author tries speculative decoding:

• Use Qwen 3.5 800M as a drafter proposing 8 next tokens
• Verify those tokens in bulk with the big model

Observed:

• draft acceptance ~65%
• expected speedup (2–4×) did not happen
• speed fell from 17 tok/s → 11 tok/s

Why it fails (architectural reasons)

1. MoE expert selection + CPU fetching

   • batching 8 tokens may touch many different experts across tokens
   • potentially PCIe thrash (e.g., many experts per layer across positions)

2. State space model (SSM) layers

   • ~30/40 layers are SSM, computing one position at a time with dependencies on previous state
   • can’t parallelize verification across the draft window like transformers can

Conclusion: speculative decoding works better for transformer-style models; it doesn’t fit this MoE + SSM architecture (consistent with a cited benchmark across many configs on a 3090).

Follow-up mentioned:

• Dflash (“block diffusion drafter”) might work better for a different/dense setup (implied chance if using a dense variant), but it’s not the direct MoE+SSM case here.

---

Final claimed “best setup” outcome

• Model: Qwen 3.6 35B A3B (MoE)
• Hardware: GTX 1060 6GB + i3-8100 + DDR4
• Result: ~17 tokens/sec
• Context: 256,000 tokens
• Stability: no long-run degradation after ~day(s), due to mlock setup
• Presented as: “five flags” + one Docker command

---

Main speakers / sources (as mentioned)

• Speaker/author: the YouTube creator narrating the guide (no name provided in subtitles)
• Primary tools/models cited:
  • llama.cpp
  • Qwen 3.6 35B A3B (target)
  • Qwen 3.5 800M (drafter for speculative decoding)
  • Google DeepMind (Turbo Quant paper)
  • Dflash (block diffusion drafter paper)