Running vLLM on Strix Halo (AMD Ryzen AI MAX) + ROCm Performance Updates

Running vLLM on Strix Halo (AMD Ryzen AI MAX) — ROCm updates

Hands-on guide and benchmarks for running vLLM on the Strix Halo (RDNA 3.5 integrated GPU / unified memory), with attention-backend comparisons, ROCm/The Rock updates, and practical configuration tips.

What the video covers

• How to run vLLM on the Strix Halo and how to configure the system for best unified-memory usage.
• A comparison of attention backends in vLLM (Triton vs AMD ROCm attention kernels), including a toggleable toolbox to test backends per model.
• Benchmarks (tokens/sec) across models and quantizations showing mixed results: ROCm attention can be much faster for some models (e.g., Gemma 3 12B) but slower or failing for others (e.g., Qwen 3 Coder/Next).
• Model compatibility and quantization limitations on Strix Halo: RDNA 3.5 lacks some instructions (no native FP8), so many FP8 or some AWQ/GPTQ quantizations won’t load natively; dequantization to FP16/BF16 is possible but removes memory savings.
• Practical guidance: run a Fedora host, tweak kernel parameters to expose nearly all of 128 GB unified memory while reserving ~4 GB for the OS, and use a Docker image built with vLLM + nightly ROCm/The Rock builds (AO Triton / ROCm attention).
• Stability caveats and the need to experiment model-by-model to find working quantizations and backends.

Key technical points and product features

vLLM

• Targeted at server-style workloads (multi-request handling, larger batches, tensor parallelism).
• Best performance on supported data-center GPUs; support on consumer AMD GPUs (Strix Halo) is still immature and model support is patchy.

Attention backends

• Triton: the safest baseline with broader reliability.
• ROCm attention (AMD native kernels): can outperform Triton for certain architectures but has uneven support on RDNA 3.5; results are use-case dependent.

Quantization and model formats

• GGML / llama.cpp: very versatile for local inference; community GGML quant options make many models easy to run and often performant on consumer hardware.
• FP8 models: work natively on newer RDNA (e.g., R9 700 AI Pro) but not on Strix Halo; dequantization to FP16/BF16 is possible but negates FP8 memory savings.
• AWQ / GPTQ variants: compatibility varies and some may fail depending on available kernels.

ROCm / The Rock

• ROCm 7 and The Rock PyTorch builds (AO Triton included) are available.
• AO Triton compiles key kernels ahead-of-time (AOT), improving stability and performance (notably for diffusion/image workloads).
• ROCm attention uses AMD’s native kernels (not Triton-generated), giving speedups for some models but introducing incompatibilities for others.

llama.cpp

• Developers added AMD-specific custom kernels, improving performance and diminishing the usefulness of earlier ROCW MMA tooling.
• For many models on Strix Halo, llama.cpp currently yields better practical performance than vLLM due to available quantizations and optimizations.

Image / video generation

• Rebuilding toolboxes on The Rock/nightly PyTorch with AO Triton can give ~2x speedups for many diffusion models (e.g., Qwen Image, ComfyUI).
• ComfyUI stability improved on recent Linux kernels.

Community and tooling

• Toolbox + benchmark scripts and a Docker image are available on the author’s GitHub.
• Active Strix Halo Discord community; AMD and contributors are iterating on kernels.
• AMD is collecting GPU kernel traces from ComfyUI users to improve ROCm stability and performance.

Practical takeaways and recommendations

• You can run vLLM on Strix Halo, but expect compatibility limits and the need to test models individually.
• Start with Triton as the baseline, and try ROCm attention for models where it may yield large gains.
• If you depend on FP8 memory savings, Strix Halo is likely unsuitable unless you accept dequantization (and the lost memory benefit).
• For many consumer use cases, llama.cpp with GGML quantizations may be simpler and faster today.
• Use the provided Docker image, benchmark scripts, and the GitHub repo to reproduce tests.
• Consider joining the Strix Halo Discord and contributing kernel traces if you run ComfyUI.

Reviews, guides, and resources mentioned

• The video: hands-on guide to running vLLM on Strix Halo, throughput benchmarks, and attention-backend comparisons.
• GitHub repo: toolbox, scripts, Docker image, and benchmark code (link referenced in video).
• Prior video: deeper vLLM explanation and a dual R9 700 comparison (recommended for more detail).
• Instructions for kernel parameter settings to expose unified memory while reserving ~4 GB for the OS.
• AMD GitHub issue / instructions for enabling GPU kernel trace dumps (for ROCm optimization, ComfyUI users).

Main speakers and sources

• Video presenter / channel author — demonstrates setup, benchmarks, and provides the toolbox.
• AMD / ROCm (including The Rock builds and ROCm 7).
• llama.cpp developers and community contributors.
• Community sources: Strix Halo Discord and ComfyUI users.