Qwen 3.6 14B A3B FableVibes benchmarked and tested vs Base Qwen 35B - 16GB Local LLM setup

Overview / what’s being tested

• The video compares a pruned fine-tune of Qwen3.6 35B A3B down to a 14B variant (labeled “Qwen 3.6 14B A3B” in the subtitles), against the base Qwen3.6 35B A3B.
• Primary motivation: local deployment
  • The 14B model fits entirely in 16GB VRAM, enabling faster inference.
• The 14B model is tested with two quantizations:
  • Q4KM (the usual baseline choice)
  • Q8 (higher precision, slower)
• Base model baseline:
  • Q4KM 35B A3B
  • Later also tests IQ4NL quantization for the base
• GitHub is referenced for model configs and benchmark documents.

Hardware / setup

• Test system:
  • 16GB VRAM
  • 32GB DDR4 RAM
• Benchmarks are designed to reflect local runtime characteristics and reasoning/coding ability.

---

Benchmarks / results (technological focus)

1) Performance: prefill vs decode speed

Definitions used

• Prefill: prompt ingestion speed
• Decode: token generation speed (output)

14B results

• Q8 (14B):
  • ~260–270 tokens/s prefill
  • ~30 tokens/s decode
• Q4 (14B):
  • ~1000+ tokens/s prefill early, dropping to ~900 at large prefill
  • ~65 tokens/s decode

Base (35B Q4KM) vs 14B Q4

• Prefill: ~180 (base Q4) vs ~900/1000 (14B Q4)
• Decode: ~37–38 (base Q4) vs ~65 (14B Q4)

Conclusion (speed)

• 14B Q4 is significantly faster, helped by fitting fully into VRAM.

---

2) Memory benchmark (“needle in a haystack”)

• Context is filled to 256K
• Target is inserted at depths: 0/25/50/75/100%
• 3 trials per depth (total 15 trials)
• Models must retrieve the correct data, testing whether long-context attention “gets lost.”

14B results

• 14B Q8:
  • Perfect at all depths (no failures)
  • Runtime: ~9h 41m
• 14B Q4:
  • Also perfect at all depths
  • Runtime: ~4h 14m

Only difference between Q4 and Q8 here: runtime, not accuracy.

Base 35B vs 14B

• Base 35B (Q4KM):
  • Similar performance overall, but had one failure at 100% depth
  • (1 out of 3 trials failed at that depth)

Conclusion

• 14B fine-tune retains long-context competence very well.

---

3) “Agency benchmark” (company sandbox with tools)

Models run in a fake company environment with:

• multiple tools
• multiple scenarios

• a multi-step tool-chaining requirement

• Benchmark version 2.0 adds more complex org structure and scenarios.

• Evaluation uses an “accuracy by skill area” style breakdown.

14B Q4 vs 14B Q8

• Q4 slightly better than Q8:
  • one more scenario correct
  • also faster
• Common 14B Q8 failure modes:
  • Calling too many tools (e.g., 7 calls vs expected 6)
  • “wrong action” mistakes (e.g., overbooking meeting rooms)
  • suggests overthinking / inefficient tool usage

Base 35B

• Similar closeness:
  • took longer
  • got one extra question correct vs Q4
  • got one extra vs Q8
• Also shows “too many tool calls” on at least one question.

Conclusion

• Agency performance was very close across models; no decisive winner.

---

4) OpenAI human eval (164 handwritten Python challenges)

• Uses OpenAI human eval dataset:
  • 164 handwritten Python tasks
• Compares:
  • pass rates
  • “questions answered” behavior

14B Q8 vs Q4

• Pass rates:
  • 56% (Q8) vs 59% (Q4)
• Ranking by answered questions:
  • ~90% answered correctly (Q8) vs 92% (Q4)
• Q8 failed to answer 62 questions; Q4 failed to answer 58.
• Note: models can run out of “thinking budget” (set to warn around half of an 8K context budget), but this doesn’t fully prevent timeouts/aborts.

Base 35B vs 14B Q4

• Base shows a clear improvement:
  • ~80% pass rate (base) vs 59% (14B Q4)
  • ~98% correctness on answered subset vs lower for 14B

Conclusion

• The base model is much stronger on Python synthesis tasks.

---

Coding challenge demos (browser-based)

Two interactive coding prompts are run in a browser harness (Python execution / JS output implied by “serve file”), then evaluated by behavior.

A) Sand physics simulator

Test order: 14B Q4, then 14B Q8 if Q4 succeeded, then base 35B.

14B Q4

• Generates a working sim scaffold (canvas/grid style).
• Partially works:
  • sand placement works
  • walls stop sand
• Issues:
  • water behavior wrong (e.g., water passes through sand incorrectly)
  • acid behavior incorrect
• Fix attempts trigger signs of thinking loops (repetitive “but wait/let me try”).

14B Q8

• Also enters a stuck loop in code regeneration/check cycles.
• After reload attempts and extra prompts: became non-interactive (no ability to place materials).

Base 35B

• Works correctly in one shot:
  • sand, water, walls, acid interactions behave as expected
• Minor artifact:
  • “white pixels” in sand (visual glitch, not physics failure)
• Performance noted: ~60 fps

Conclusion

• Base is far superior on this task.

---

B) Dungeon crawler / raycasting

Test order: 14B Q4, 14B Q8, then base 35B, then base with improved quantization.

14B Q4

• Movement issues:
  • laggy/inconsistent inputs depending on key holding and map updates
• “God mode” updates position correctly, but normal interface updates are wrong → suggests state/position synchronization bugs.
• Raycasting incomplete in places.
• Controls/map behavior still inconsistent.

14B Q8

• Controls severely jumbled (W/S/A/D mapped incorrectly).
• Raycasting visually “not too bad.”
• Fog-of-war / explored areas not working.
• Also shows thinking-loop when trying to fix issues.

Base 35B

• Movement works.
• Raycasting partially imperfect (corner artifacts), but generally functional.
• Fog-of-war / explored areas better than 14B.
• After prompting to improve raycasting: looked about the same.

Quantization experiment on base model

• Switching base quant from Q4KM to IQ4NL:
  • described as “everything working perfectly”
  • fog-of-war/explored behavior works
  • raycasting behaves correctly during movement

Conclusion

• For this base-model test: IQ4NL > Q4KM.

---

Final conclusions claimed by the creator

• Overall winner: base Qwen 3.6 35B A3B, especially on code-heavy benchmarks.
• Speed trade-off:
  • 14B Q4 is extremely fast, but the intelligence drop shows up on coding tasks.
  • Q8 is slower and often doesn’t provide enough accuracy benefit to justify the cost.
• Memory benchmark:
  • 14B performs extremely well on long-context retrieval.
• Agency benchmark:
  • models are close; no strong separation.
• Human eval + coding:
  • base model clearly wins, including interactive coding tasks.
• Quantization matters:
  • For the base model, IQ4NL was highlighted as the best quant choice in the final dungeon crawler test.

---

Main speakers / sources

• Speaker: Luke (host of “Luke’s Dev Lab”)
• Referenced dataset/source: OpenAI human eval (164 handwritten Python challenges)
• Referenced model baseline: “Unsllo” in the subtitles (likely the author/model source for Qwen 3.6 35B A3B Q4KM)