Qwen3.8 27B Is INSANE — 108 Tok/s on One RTX 5090

Overview / Key release claims

• The video focuses on Qwen 3.8 with 27B parameters (a dense model) as a highly anticipated release, alongside another larger Qwen 3.8 variant (~2.4T parameters, mentioned).
• The creator tests multiple BF16 / quantized versions of Qwen 3.8 and emphasizes practical performance on consumer hardware.

Hardware + model sizing + runability

• The creator demonstrates running Qwen 3.8 27B on a single RTX 5090 (32GB VRAM).
  • Other loaded assets consume about ~8GB VRAM, but the model can still run with roughly ~24GB VRAM available.
  • The creator claims it can also fit on 12GB VRAM when using quantization.

Quantization size targets

• Q4: model around ~17GB
• Q2K XL (2-bit quant): around ~10GB

Context length

• Uses about ~131K context on the RTX 5090.
• Suggests it may reach ~266K context if VRAM is managed carefully.
• Notes that spilling into system RAM significantly slows performance.

Throughput / decoding speed improvements

• The main performance driver is speculative decoding / multi-token prediction.
  • The model predicts 2–4 tokens ahead, then verifies them “all at once.”
  • Reported speedup: about ~2.5× to ~3× in practice.

Reported decode / throughput numbers

• Around ~108 tokens/sec decode on RTX 5090 (for the described scenario).
• Mentions speculative support such as DeepFlash v2 and Unsloth Studio features “coming shortly,” potentially approaching ~200 tokens/sec decode.

Latency concern addressed

• The model “thinks a lot” (long outputs / tool usage), but the creator reports very fast loops and says it’s not worried about latency.

Accuracy / coherence vs quantization issues

• The creator argues quantization is becoming less of a dealbreaker for agentic workflows.

Traditional quantization problem (described)

• Quantizing (e.g., FP16/FP32 → Q4) can introduce token errors (e.g., “wrong token” happening at some fraction of lines).
• In non-agent settings, those errors can cause:
  • looping
  • non-coherence
  • broken outputs (example given: a misplaced element in an image artifact)

Agentic mitigation (claimed)

• With agentic frameworks, the model can:
  • detect and correct after the fact
  • use tool calls / verification loops
• This helps preserve final quality even if some token-level mistakes occur.

Tool use / agentic verification workflow (Unsloth Studio)

• The video demonstrates generating complex visuals (example: an “Acropolis” scene).
  • Outputs are described as being generated in JavaScript/HTML.
  • The creator claims earlier local-model workflows were difficult, but Qwen 3.8 27B can reach ~90–95% of the quality of much larger models for that specific task.

Verification and correction loop (example workflow)

• The model performs tool calls directly in Unsloth Studio to validate outputs.
• Example workflow described:
  1. Generate an SVG/image-like output (example includes: horse on a blue bicycle, camel in background; one prompt included a typo “bicycle”).
  2. Read the produced file, verify coordinates with Python, and update coordinates if needed.
  3. For colors:
     • instead of direct vision, it imports a PNG
     • converts it into a color table
     • edits based on that analysis

Iteration speed

• “Loops and iteration loops” happen quickly on the 27B model.
• Example indicates image/art refinements at roughly ~100 tokens/sec after prompt processing.

Benchmark / comparative analysis claims

• The creator references the Artificial Analysis Index:
  • Qwen 3.8 27B shown at ~52 (with the note that benchmark scoring can shift over time).
• Interpretation described as XHi vs Low:
  • Medium drops more than Low, but Low still performs relatively well.
  • Suggests selecting XHi or Low depending on needs.

Positioning vs other models

• Qwen 3.8 27B is positioned as competing with models including:
  • Llama / “Luna” GP 5.6
  • DeepSeek V4 Pro Max
  • (and others mentioned)
• The creator claims Qwen 3.8 27B is a couple points away from “frontier” performance in that index.

Qualitative conclusion

• Even though it’s much smaller than frontier models, it’s described as:
  • close enough on the benchmark
  • strong enough in coherent generation + agentic correction

Main speakers / sources

• Main speaker: the video author/host (referred to as “Hey guys…”; no name given in the subtitles)
• Primary source/model under discussion: Qwen 3.8 (27B)
• Tooling/workflow environment mentioned: Unsloth Studio (including references to DeepFlash v2 support)
• Benchmark referenced: Artificial Analysis Index