Best Local Coding AI for Your GPU (4GB to 512GB)

Key technological concepts (why “it fits” isn’t enough)

• GPU memory isn’t just model weights. Charts often show only weights, but VRAM/RAM must also hold:

  1. Weights (the downloaded model file)
  2. KV cache (stored “notes” for the prompt + generated tokens; grows with context length)
  3. Scratch/runtime workspace for computation

• If weights + KV cache + scratch exceed available memory, the model either:

  • Spills to slower memory (hurting speed), or
  • Crashes (failing to run)

• When memory is tight, you have two main dials:

  • Reduce quantization level (use fewer bits per weight → smaller weights)
  • Trim context (shorten prompts / conversation history to reduce KV cache)

• Even if a bigger model loads, it may be less useful if it leaves insufficient room for:

  • long files,
  • multi-step planning, or
  • agent loops

---

Product/model tier guidance (what to run for different GPU memory sizes)

~4 GB VRAM (entry-level “coding assistant,” not full project work)

• Example model: SparkX 2.5 (4B), ~2.6 GB quantized weights.
• Needs runtime support for its specific attention design (may not load if runtime isn’t compatible).
• Best for:
  • explaining code you didn’t write,
  • fixing a single focused bug,
  • small edits
• Limitation: struggles with multi-file reasoning/planning

~6 GB VRAM (small quality bump)

• Example model: Neo Horse 1 (4B) around ~3 GB at higher quality settings.
• Built on Qwen 3.5, tuned for tool use in coding; improves on agentic/tool benchmarks relative to base.

~8 GB VRAM (first “real step” into 9B models)

• Example model: Ornith 1.5 (9B) ~5.75 GB at 4-bit.
• Positioned as moving from autocomplete-like behavior toward a junior pair-programmer.
• Constraint: keep prompts lean to preserve context budget.

~12 GB VRAM (common mistake: choosing a bigger model instead of better quantization)

• Instead of going from 9B to a larger model, the guidance suggests raising quant quality:
  • Same 9B model at 8-bit (~9.75 GB) can be steadier/cleaner
• Quantization guidance:
  • 4–5 bits is described as a “safe zone”
  • <3 bits quality can drop sharply (1-bit is treated more as a novelty)

~16–24 GB VRAM (sweet spot for multi-file edits)

• Target: run larger models with enough headroom for longer context.
• Example high-memory method:
  • Qwen 3.8 27B in a special build using smarter quantization to fit under ~16 GB while preserving benchmark quality
  • Uses precision sensitivity / tensor-wise decisions (more bits where sensitive, fewer elsewhere)
• At ~24 GB, enough room for multi-file changes + test suite reasoning.

~32 GB VRAM (two choices; “bigger” isn’t always best)

• Example options:
  • 27B at higher quality (e.g., ~6-bit, ~22 GB)
  • Ornets 35B around ~25 GB

• Key technical point: Mixture of Experts (MoE) means only a subset of parameters activate per token (e.g., ~3B active out of 35B total).

• Important caveat:

  • Sparsity helps speed/efficiency, not necessarily memory fit (you still need to store the full model in memory)

• Quality can feel less consistent on the hardest reasoning despite large headline size.

~48–128 GB VRAM (coding agents and real iterative loops)

• Example workhorse: Code 3 Coder Next (80B) (quantized variants depending on VRAM):
  • ~48 GB: ~38 GB at 4-bit
  • ~64 GB: ~48.5 GB at 4-bit
  • ~80 GB: ~66 GB at 6-bit
• Agent loop described:
  • read failing error → edit file → run tests → read new error → continue,
  • maintaining a long task context
• Bottleneck warning at this tier:
  • Sometimes memory bandwidth / bus latency becomes limiting when offloading layers to system RAM (GPU waits for data)
• Multi-card nuance:
  • H100 NVL (data center) vs RTX Pro (workstation) differences exist, but at scale interconnect bandwidth dominates.
  • Cards with different VRAM sizes can behave differently due to links.

~141 GB and up (frontier; architecture + runtime compatibility becomes a blocker)

• Example family: GLM 5.3 Flash (and full GLM 5.3):
  • roughly 120–240 GB across quant levels for the “Flash” model
  • full model up to ~467 GB
• Critical constraint:
  • The architecture may not yet be supported in the mainline runtime
  • A working setup may require:
    • a specific Unsloth-maintained branch, or
    • the desktop app
  • Otherwise, weights download but “nothing runs”.

~256 GB+ and frontier comparison

• Example: MiniMax N3 (426B) described as more practical than “obviously bigger” models because it’s also MoE-like (only ~23B active per token).
• Tradeoff described:
  • Larger dense models may “eat” too much memory, leaving little for long context
  • A sparser model can keep speed + headroom

---

Experimental approach: running parts from SSD (hierarchical memory)

• Example: DeepSeek v4.1 Flash (552B) with DwarfStar (Mac engine).
• Idea:
  • Keep a portion (lookup/table-like structure) persistently on SSD, streaming it as needed.
• Benefits:
  • enables running compressed versions on ~128 GB Mac
  • quality increases can expand the main model while the on-disk component stays fixed-ish
• Framing:
  • “frontier direction” is tiered/hierarchical memory
    • hot parts in fast memory
    • cold parts on disk
• Caveat: early/finicky, not reliable for deadlines.

---

Practical “call” (what the guide recommends most viewers do)

For most people with ~8–24 GB VRAM, pick the model that leaves headroom, not the one that barely loads.

• Suggested examples:
  • 9B in 8–12 GB
  • 27B in 16–24 GB
• For long-running agentic work, the 48–128 GB tier earns its keep.

Memory fit is the floor, not the finish line—final success depends on:

1. Model file
2. Runtime compatibility
3. Context length
4. Time to reach correct results
5. Where it failed (not just whether it loaded)

---

Main speakers/sources (as implied by the subtitles)

• The video narrator/host: gives hardware-and-model recommendations (no specific person named in the subtitles).
• Unsloth team: runtime support and maintained branches.
• Model repositories/teams referenced:
  • Qwen
  • GLM (and MiniMax)
  • DeepSeek
  • Code 3 (Coder Next)
  • plus a Mac engine called DwarfStar