Video summary

Qwen3.6-27B on M5 Max: 4-bit vs 8-bit vs BF16 Benchmarked

Main summary

Key takeaways

Technology

Video Summary: Qwen3.6-27B on Mac (MLX) — 4-bit vs 8-bit vs BF16 Benchmarks

Goal / Setup

  • Benchmarks Qwen3.6 27B (latest Alibaba release) running on Mac using MLX.
  • Compares quantization levels: 4-bit, 8-bit, and BF16.
  • References an inference engine called OMLX, including a mention of a prior tutorial for it.
  • Uses a “server” UI panel to configure and run benchmarking, including:
    • model selection
    • prompt evaluation
    • context length
    • batching settings

Mac Hardware

  • The video references a specific Mac model (“Here is the Mac spec I’m using”), but the actual specs are not shown in the subtitles.

Benchmark Tool / How Testing Is Done (UI Features)

In the benchmarking interface:

  • Select model variant
    • e.g., 4-bit / 8-bit / BF16
  • Single request mode
    • choose prompt / prompt evaluating
    • adjust context length (examples include 15k and higher)
  • Continuous batching mode
    • runs parallel sessions by increasing batch size (e.g., 2x, 4x)
  • Tracks benchmark metrics including:
    • GPU utilization (often reaches ~100%)
    • Unified RAM usage
    • Tokens per second (tokens/s) split into:
      • Prompt evaluation (processing the prompt)
      • Text generation

Results: 4-bit Quantized

Model Size / Memory

  • Uses about ~26.7 GB unified RAM.

Performance Behavior

  • GPU utilization reaches about ~100%.
  • Increasing context length reduces speed:
    • At 16,000 context length, generation drops to ~17.5 tokens/sec
    • Prompt evaluation drops to ~861.9 tokens/sec (as stated)

Continuous Batching (Parallel Sessions)

  • 2 sessions: ~23.1 tokens/sec
  • 4 sessions: ~40 tokens/sec
  • Key takeaway: throughput improves as batch size / parallelism increases.

Results: 8-bit Quantized

Model Size / Memory

  • Uses about ~33.9 GB unified RAM (roughly 10 GB more than 4-bit).

Performance Behavior

  • GPU utilization again reaches about ~100%.
  • Slower than 4-bit.
  • Reported generation speed while varying context ranges roughly:
    • ~15 → 13.7 tok/s
    • (trend varies slightly across tested lengths)

Continuous Batching

  • Shows the same general trend: higher batch size improves speed.

Results: BF16 (Highest Precision)

Model Size / Memory

  • Model footprint is about ~53.5 GB.
  • Hardware requirement mentioned:
    • Mac should have at least 64 GB unified RAM

Performance Behavior

  • GPU utilization is about ~99–100%.
  • Significantly slower than 4-bit and 8-bit.
  • Generation speed decreases as context increases:
    • ~9.5 → 9.4 → 8.9 → 8.7 tok/s

Continuous Batching

  • Still benefits from increased batch size (speedups observed, consistent with other variants).

Overall Conclusions (Implied by the Benchmarks)

  • 4-bit is the fastest in these tests, with the best generation throughput and strong scaling under continuous batching.
  • 8-bit requires more RAM than 4-bit, with slightly lower throughput.
  • BF16 needs the most memory and is the slowest, though it still improves with continuous batching.
  • Across all variants:
    • GPU utilization tends to saturate near maximum
    • Longer context reduces generation speed
    • Higher parallel/batch size improves throughput

Main Speakers / Sources

  • Video narrator / channel host (name not given in the subtitles)
  • Mentions OMLX and MLX on Mac (platform/tools referenced)

Original video