Video summary

Run Your Own AI Coder on a 6GB GPU (No Subscription)

Main summary

Key takeaways

Technology

Technological focus / main analysis

  • Goal: Test whether small local LLMs (that can run on a 6GB GPU) can complete real coding tasks, not just short “chat” prompts.
  • Hardware assumption: Runs fully on one machine (not cloud chat). Example setup: 16GB RAM + 6GB GPU.
  • Core issue: On a 6GB card, model file size on disk is not the same as GPU memory required to run.

The video highlights the difference between:

- **Download size / weights size**
- **Runtime memory** needed for **context cache** + working space

Key decision metric: GPU-resident memory

  • The metric that matters is: how much must reside in GPU memory at once, after accounting for:
    • desktop/driver overhead
    • runtime caches
  • Rough planning figure: ~5GB usable GPU memory (after desktop + drivers)

Context cache impact

  • Context cache (short-term memory) scales with context tokens:
    • ~4,000 tokens: cache stays relatively small
    • ~8,000 tokens: cache roughly doubles

Bottom line: Memory math determines feasibility before you even worry about downloading the model.

Models discussed (four to try on a ~6GB GPU)

  1. Neo Horse 1 – 4B, 5-bit

    • About 3GB on disk
    • Designed for coding instructions and running as an agent
    • Positioned as the “practical starting point”

  2. Sparkx 2.5 – 4B, 4-bit

    • About 2.6GB on disk
    • Claimed improved usability: its page lists native support in mainstream tools (less special runtime hassle)

  3. Neo Horse 1 – 9B, 4-bit

    • About 5.6GB on disk
    • Likely not fully fit once context is added
    • Expects layer offloading to CPU RAM (“slow path”)

  4. Bonsai 2 – 27B (compressed), ~5.95GB download

    • Download may fit on disk, but:
      • requires Prism lab “patched runtime” (normal tools reject the file)
      • compression doesn’t guarantee it will fit comfortably in actual runtime GPU memory
    • Treated as unproven for full 6GB residency

Setup / product tooling (tutorial-like stack)

  • Model runner: LM Studio

    • Loads model files and serves them
    • Provides crucial controls/dials:
      • Context size
      • Offload split (how many layers remain on GPU vs move to CPU)
    • Includes a memory estimate check before loading (important for 6GB planning)

  • Editor: VS Code

  • Integration extension: Continue
    • Connects VS Code to the model running in LM Studio
    • Enables an iterative coding loop using tool calls (read files, write edits, run tests)

Why this stack matters

  • The argument is that real coding requires a read → edit → test loop (agent/tool execution), not just chat-style suggestions.

Practical configuration guidance emphasized

  • Use one model at a time so it has the card to itself.
  • Start with:
    • Context = 4,000 tokens
    • Try 8,000 only if LM Studio’s memory check says it fits
  • Feed only what matters:
    • relevant files and test output, not the entire repo
  • Use model settings recommended by its page:
    • Reasoning mode
    • add a hard cap (time/amount allowed to write) to avoid infinite thinking
  • Rehearse before real work: run a small rehearsal (one file, one small change, one test) to verify the read/edit/test loop works.

Versioning / compatibility notes (footnotes)

  • Sparkx: needs recent tool builds (around early September releases or newer) or it may not work.
  • Bonsai: requires Prism patched runtime; stock tools won’t load it.

Evaluation method (three “jobs” on a real broken codebase)

Not a leaderboard—this is real work on a broken codebase (including a web API failing with HTTP 500).

Job 1: Fix the broken API (memory feasibility + correctness)

  • Provide:
    • root file
    • failing test output
  • Watch whether the model can:
    • hold file + test context simultaneously
    • produce a change that makes the test pass
  • Framed as the fairest test of whether a 4B model can actually code.

Job 2: Add a feature (context stress + coherence over multiple files)

  • Harder because it requires:
    • touching more files
    • tracking more of the codebase
  • Expected failure mode:
    • as context grows, the model may contradict itself
  • The 9B model is expected to show a slow-path penalty here.

Job 3: Repair its own mistakes (agent reliability)

  • Start with the model’s messiest output:
    • feed the new failing test back in
    • ask it to fix what it broke
  • Emphasis:
    • benchmarks can look good on single-shot tasks, but this tests recovery after a second failure.

Key findings / conclusions (what to use)

  • Best “daily driver” on a 6GB GPU: choose between models that fit with headroom:
    • Neo Horse 1 4B (5-bit)
    • Sparkx 2.5 4B (4-bit)
  • 9B Neo Horse 1 (4-bit): treat as a weekend/experiment
    • likely spills to CPU (slower), trading speed for potentially better answers
  • 27B Bonsai 2: treated as a lab curiosity, not a reliable 6GB setup
    • file size can be misleading (download vs runtime needs)
    • requires patched runtime and is not proven to fully run in 6GB GPU memory

Main speaker / sources (as shown in the subtitles)

  • Main speaker: the video creator / narrator (no specific name provided in subtitles)
  • Product/software sources mentioned:
    • LM Studio
    • VS Code
    • Continue (VS Code extension)
    • Prism Lab (patched runtime for Bonsai)
  • Model sources mentioned:
    • Neo Horse (Neo Horse 1, 4B and 9B)
    • Sparkx 2.5
    • Bonsai 2

Original video