Claude Grade Coding From An 8.4GB Local AI Model

Main technological ideas / product details

Model quantization (shrinking)

• A 27B-parameter Quen 3.8B model is normally stored at full precision (16 bits/number), producing a ~53.8 GB model.
• Quantization reduces size by storing weights with fewer bits per number, roughly ~2.5 to 3.5 bits on the smaller files.
• This is likened to lowering image quality: smaller files lose precision and can degrade output quality, especially on harder tasks.

Distributed model files

The lab provides multiple quantized weight files of the same underlying model:

• 8.4 GB, 9.3 GB, 10.1 GB, 11.8 GB (text weights)

There is also a separate ~0.9 GB vision file at 16-bit precision to enable image understanding.

Compatibility / deployment

The lab claims these quantized files run unmodified in tools like Llama.cpp / LM Studio, allowing the model to fit on a single consumer GPU rather than requiring server memory.

---

What the benchmarks actually mean (and where the hype comes from)

“100.3%” claim is not “coding better”

• The 8.4 GB file averages 100.3% vs the full model on a set of five multiple-choice quizzes (zeroshot).
• However, the video emphasizes these quizzes mostly measure probability ranking, not genuine generation or step-by-step coding.

Real coding tests show a sharper drop

On Live Codebench v6 (hidden tests; code must run/pass):

• Full model: 85.7
• 11.8 GB: 85.7 (no visible drop)
• 8.4 GB: 76.6 (≈ -9 points)

On GPQA Diamond (hard science):

• Full: 89.9
• 8.4 GB: 84.9 (≈ -5 points)

Key takeaway from lab results: Code quality degrades the most at the smallest size; the “100.3%” multiple-choice score doesn’t reflect real coding ability.

---

Independent verification vs lab claims (mixed verdict)

Byte Smith / Byte Shapes external testing

• 11.8 GB external tests: “practically matches” the full model.
• 8.4 GB external tests: only ties an ordinary similarly-sized Unsloth file, rather than outperforming it as lab tables suggest.
• Disagreements between lab and outside results are presented as evidence the smallest file isn’t as “magical” outside lab conditions.

---

Language sensitivity / practical failure case

A lab discussion anecdote:

• A user tested Chinese prompts on a mid-small file (10.1 GB) and got broken code that wouldn’t execute, even after retries.
• When the prompt was translated to English, the problem disappeared.
• The 11.8 GB file performed noticeably better.

Explanation given:

• About 90% of compression-tuning text was English, so non-English performance drops more.

---

“Claude Grade Coding” comparison: who wins and why it may not translate

Video framing vs what the comparisons really cover

The video argues Claude comparisons apply to the full Quen model, not the 8.4 GB quantized file.

On Quen’s model card, comparisons use different setups/bench variants:

• Example terminal/agent coding: Quen trails Claude (e.g., 73 vs 78.2).
• Live Codebench contest: Quen can edge out Claude in their setup (90.3 vs 88.8).
• Benchmark and tool choices make direct comparisons difficult.

Independent study findings (August)

• Single-prompt coding: Quen averages ~63, Claude ~87.4.
• Quen improves only under a manager setup that increases planning and uses far more compute.

Compute/token cost

Manager setup increases tokens dramatically:

• Claude: ~1.2M output tokens (single calls)
• Quen: ~7.4M tokens
• Manager setup: ~18.6M tokens to tie

The video frames this as a practical downside: more tokens → slower response / more waiting.

Key point: none of these Claude-vs-Quen studies evaluated the 8.4 GB file directly.

---

Guide: which quantized file to download for coding (based on GPU size)

• 16 GB GPU: download 11.8 GB
• 12 GB GPU: choose 10.1 GB (more capable than 8.4 GB)
• 8 GB GPU: only consider 8.4 GB as a fallback (small enough to fit, but limited)

Important nuance: memory isn’t just the model file

• The vision file (~0.9 GB) can consume almost another gigabyte.
• Conversation/cache memory matters:
  • Estimated at about 64 KB per token (full precision)
  • Rough sizing:
    • ~16,000 tokens per GB at full precision
    • ~32,000 tokens per GB when cache is compressed to 8 bits

On 12 GB GPUs, loading vision can leave too little memory for longer chats with the 11.8 GB file—so the practical choice becomes 10.1 GB vs 8.4 GB.

Trade-offs summarized (12 GB scenario)

• 10.1 GB
  • Coding benchmark: 84.6
  • Leaves < 2 GB memory “breathing space”
• 8.4 GB
  • Coding benchmark: 76.6
  • Leaves ~3+ GB, roughly double the conversation runway

Speed comparison

On Byte Shapes:

• 8.4 GB: ~60.5 tokens/s on a 16 GB card
• 11.8 GB: ~48.4 tokens/s

The video claims the speed gain (~25%) isn’t worth losing about ~9 coding points.

---

Practical “how to use” recommendations from the lab discussion

• Prefer English prompts (compression tuning was mostly English).
• Keep reasoning turned on (per the lab member).
• Use the 8.4 GB file as an emergency spare, not a daily driver—because it’s the weakest coder and performs worse outside lab conditions.

---

Main speakers / sources (as referenced)

• Alireza Dadgara (lab researcher; comment explaining benchmark interpretation)
• IstA Das Lab (the lab/research team referenced for quantization + benchmark tables)
• Byte Shapes / Byte shape (external evaluator; compressed model comparisons)
• Independent study authors (August study referenced for Claude vs Quen comparisons)