Video summary

Wait, you can run GLM-5.3 Flash and Deepseek V4 Flash at home for $800!?

Main summary

Key takeaways

Technology

Summary of the video (tech concepts + local running vs subscriptions)

Model release & identity

  • ZpU released GLM 5.3 Flash, which the speaker says is the previously “mystery” oX/Alpha MoE model tested anonymously on platforms like Open Router/OpenCode.
  • Weights are on Hugging Face and are MIT licensed.
  • Claim (with a caveat): it beats GLM 5.2 on benchmarks while costing about ~1/10; coding/agentic performance is said to approach Claude Opus level. Benchmarks are advised to be taken “with a grain of salt.”

Key architecture / why it matters for running locally

Mixture-of-Experts (MoE)

  • 320B total parameters, but only 18B active per token (the speaker calls this the “magic number”).
  • The model is described as the first natively multimodal model in the GLM 5 line:
    • Handles text, images, video, and visual documents.

Long context

  • ~1M token context window.
  • KV cache reduced ~4.4× vs a larger GLM 5.3 variant to better fit long contexts in limited memory.

Local feasibility depends on memory, not raw compute

  • You must store all 320B parameters in memory, even though only a subset is computed per token.
  • The script emphasizes a hybrid strategy:
    • Keep some layers on CPU and others on GPU to make it workable.

Quantization and practical hardware requirements

Approximate weight sizes

  • Full precision weights: ~640 GB (not realistic for home).
  • Quantized weights (approx):
    • Q8: ~340 GB (ignored as impractical)
    • Q4KM: ~190 GB (recommended “sweet spot”)
    • IQ3: ~145 GB
    • IQ2M: ~110 GB (notably usable via dynamic quant on big MoE)
    • 2-bit dynamic: ~80 GB (most aggressive; smallest)

“Entry ticket” (low-bit + offloading)

  • About 96–128 GB combined RAM + VRAM for a usable setup.
  • About 200+ GB if you want to run Q4 properly.

Offloading approach (llama.cpp)

  • Using llama.cpp with MoE offload.
  • Mentions using a flag like --n_cpu_moe to park expert layers in system RAM while:
    • shared layers + KV cache remain on GPU.

2026 cost analysis (how expensive memory is)

  • The speaker claims RAM prices surged ~500% in 12 months, driven by AI data center demand.
  • Examples cited:
    • 128GB DDR5 kits reportedly around $3,399
    • 64GB DDR5 around $1,000
    • Used RTX 3090 around $1,000 (previously a $600 bargain)
  • Conclusion: many “cheap AI server build” ideas from 2024 are no longer realistic.

Three suggested local build paths (minimum spend to comfortable)

  1. ~$800–$1,200 “DDR4 loophole” (cheapest)

    • Use older used DDR4 server/workstation ECC memory.
    • Example parts logic:
      • Used 128GB DDR4 ECC: $200–$300
      • Used Xeon/Epyc workstation platform: $400–$600
      • RTX 3060 12GB: ~$200 for shared layers/prompt processing
    • Target: 2-bit quant + MoE offloading
    • Expected speed: ~5–10 tokens/sec (slow, but “runs at home”).

  2. ~$2,500–$3,400 “unified memory box”

    • Strix Halo mini PC (Ryzen AI Max + 395) with 128GB unified memory
    • Expected speed: ~10–20 tokens/sec
    • Framed as quieter and more “plug-and-play.”
    • Enough memory to fit 2-bit quant while leaving room for context.

  3. ~$3,700–$5,600 “comfortable”

    • Mac Studio (M4 Max, 128GB): ~$3,700
      • Claimed faster due to ~2× memory bandwidth
    • For Q4 quant (~200GB+ requirement):
      • Mac Studio M3 Ultra with 256GB unified memory: mid $5,000s
    • Comparison note:
      • Nvidia DGX Spark (~$4,700) is mentioned as having the CUDA ecosystem, but it reportedly has only 128GB and less bandwidth.
      • The speaker prefers the Mac option.

Electricity cost

  • DIY inference all day: $10–$25/month
  • Mini PCs: $3–$8/month

Subscription/API cost comparison (the “brutal” part)

API pricing (GLM coding / token-based)

  • 15¢ per million input tokens
  • 50¢ per million output tokens
  • Cached input: 3¢ per million
  • Example: a “heavy day” like 20M cached input + 1M output costs under a dollar.

GLM coding plan pricing (monthly)

  • Light: $18/month (or ~$12.60/month yearly)
  • Pro: $72/month
  • Max: $160/month
  • Light tier usage (as stated): roughly 80 prompts per 5-hour window in tools like Claude Code, Kline, RuCode.

Economic conclusion

  • Using the annual Light plan (~$13/month):
    • $1,000 DDR4 build ≈ ~6.5 years
    • $3,300 Strix Halo ≈ 20+ years
    • 256GB Mac ≈ 35+ years
  • Add electricity and tinkering time.
  • Key verdict: subscription/API wins economically on pure cost, especially for “light users.”
  • Local “almost never” breaks even before hardware becomes obsolete.

Why run it locally anyway (three reasons)

  1. Privacy: code/data stays on your machine.
  2. Control/ownership: MIT-licensed weights; no rate limits/quotas/nerfs; “no rug pull.”
  3. Performance/novelty: owning a 320B multimodal model at home felt like sci-fi previously.

Recommendation stated

  • If asking “which is cheaper”: subscribe Light (~$18) or use the API (“for pennies”).
  • If you already have 128GB: download 2-bit quant to try (free).
  • Buy hardware only if privacy/sovereignty is the real priority—especially given the speaker’s 2026 RAM-price concerns.

Main speakers/sources

  • Speaker/host: the unnamed narrator of the video (no other person appears as a distinct speaker).
  • Referenced sources: Tom’s Hardware (RAM price tracking) and ZpU (model claims/specs), plus Hugging Face (weights hosting) and llama.cpp (MoE offload approach).

Original video