Video summary

From Tiny Chips to Server Racks: Deploying AI Everywhere with Rust - Stephan Eckes

Main summary

Key takeaways

Technology

Summary of technological concepts, product features, and analysis

Talk goal and scope

  • The talk focuses on deploying AI/ML models “everywhere” using Rust, spanning tiny devices to server racks.
  • It provides a landscape overview of:
    • training vs inference frameworks
    • inference engines
    • model formats
    • Rust ML frameworks
  • It includes real-world experience from a voice/speech enhancement company (AI Acoustics) working to ship real-time, low-latency model inference for clients.

Speaker background (source of real-world constraints)

  • Stephan Eckes (main speaker) comes from audio + AI engineering and open-source work.
  • At AI Acoustics, the core problem was that voice assistants/conversation agents get interrupted by background noise.
  • Their models perform speech enhancement / noise filtering, transforming spectrograms from:
    • “noisy speech” → “clean speech”
  • This improves downstream speech-to-text word error rate (WER).
  • They needed an SDK delivered via compiled, embeddable libraries, integrated into real-time systems such as:
    • LiveKit (audio pipeline; first-party plugin mentioned)
    • Pipecat (AI agent framework)
    • Other media/stream ecosystem integrations (including Stream Deck-related hardware mentions)

Training vs inference: why the tooling differs

  • Training frameworks emphasize flexibility (run-time experimentation), e.g.:
    • PyTorch
    • TensorFlow (mentioned as less prominent)
    • PaddlePaddle (PED mentioned)
  • Inference engines emphasize speed, size, and compilation to deployment targets, often relying on a C++ ecosystem with performance-focused interfaces for devices/CPUs/GPUs/microcontrollers.

Inference engines landscape (and why it’s fragmented)

  • Examples mentioned:
    • llama.cpp
    • ONNX Runtime
    • LightRT (noted as rebranded reference)
    • Executor (Meta “ExecuTorch” / executorch referenced; described as promising but sometimes not fully mature)
  • Fragmentation occurs because inference depends on:
    • model formats
    • required operators
    • quantization
    • platform-specific backends
  • Resulting in:
    • many vendor-specific backends
    • different supported quantizations
    • many target ecosystems: ARM, Apple, Intel, Qualcomm, etc.

Model formats: key evaluation criteria

Common model/weight formats and how they were characterized:

  • ONNX (Microsoft)
    • Upside: broad framework support; “fully standalone” graph+weights
    • Downside: larger/slower loading since the runtime rebuilds the graph
  • GGUF (common for LLMs)
    • Emphasizes quantization (down to 2-bit)
    • Often needs an inference engine that implements it (not fully standalone)
  • SafeTensors (Hugging Face)
    • weights only (no graph)
    • Upside: “no code injection” because it’s plain data; safe and Rust-friendly
  • PT / PyTorch weights
    • weights-only; requires accompanying model code
  • Other formats (e.g., QDQ), implied via conversion needs

Practical guidance: selecting an inference engine depends on:

  • model format compatibility
  • operator/layer support
  • quantization support
  • target platform constraints
  • performance requirements (fastest vs “good enough”)

How to evaluate an inference engine (decision checklist)

Stephan frames evaluation as:

  1. Does the engine support the model format you have (or can you convert it)?
  2. Does it support all required operators/layers?
  3. Does it support the model’s quantization method?
  4. Will it run on your target platform (CPU/GPU/NPU/microcontroller, including vendor backends)?
  5. Is latency/performance a hard requirement or flexible?

Rust ML frameworks covered (capabilities + tradeoffs)

1) art

  • Rust wrapper around ONNX Runtime (C++ backend).
  • Upsides
    • mature; runs many ONNX models
    • simpler Rust API than raw ONNX Runtime
    • supports multiple execution providers
    • provides pre-compiled binaries, enabling “cargo add and run” across many targets
    • static-by-default delivery simplifies client integration
  • Downsides/limitations
    • still depends on large C++ underneath (build/link complexity can remain)
    • prebuilt coverage may depend on newer OS/toolchain versions
    • performance advantage is model-dependent (not always faster than default ONNX Runtime)

2) burn (pure Rust)

  • Training + inference in Rust.
  • Import
    • can import SafeTensors and PyTorch/PT
    • has compile-time ONNX import (described as not perfect yet)
  • Backends
    • CPU and GPU (e.g., CUDA, HIP/ROCm, Metal, WebGPU)
  • Upsides
    • strongly Rust-native; cross-platform ambitions (including web)
    • supports programming models from scratch; forward pass resembles PyTorch style
    • provides a workflow to generate code from ONNX (when supported)
  • Downsides emphasized
    • can be slower than specialized engines (e.g., llama.cpp), depending on the model
    • no NPU support yet
    • no quantization support (called out as a big issue for LLM deployment)
    • ONNX import limitations for complex models / missing layers

3) candle (Hugging Face ecosystem; originally from Var?)

  • Pure Rust inference/training.
  • Supports
    • SafeTensors / PT weights
    • GGUF import for quantized LLMs
  • Upsides
    • well-suited for LLMs where GGUF/quantization matters
    • PyTorch-like layer/forward programming style
    • supports GPU selection (e.g., choosing device index)
  • Downsides
    • limited backends in the described setup: mainly CPU, CUDA, web
    • “alpha”/limited ONNX import operator coverage

4) tract

  • CPU-only Rust inference engine.
  • Features
    • imports many ONNX models; “just worked” for tested models
    • strong CLI tooling for inspecting layers + benchmarking + profiling (e.g., timing per layer)
    • examples included CPU benchmarking (e.g., MobileNet V2 with measured inference time and layer profiling)
  • Downsides
    • no GPU/NPU support
    • quantization status described as needing verification for specific needs

5) Other Rust inference frameworks mentioned (briefly)

  • micrflow
    • tiny, MCU-oriented; limited layer support
  • dfdx
    • compile-time shape checks; uses nightly features (interesting but possibly abandoned)
  • OpenVINO rust bindings (Intel project)
    • better fit for Intel hardware; potentially NPU on Intel devices
  • CoreMLRS
    • experimental bindings for Apple hardware
  • These “might not fit” AI Acoustics’ requirements, including:
    • real-time latency
    • allocation-free forward (for safety)
    • specific audio signal ops like STFT/ISTFT

Deep-dive into evaluation for real-time + audio workloads

Generic inference frameworks can fail for the team’s needs because they require:

  • shortest latency
  • no allocations in forward for real-time safety (especially for audio processing)
  • precise control over bottleneck layers (optimize the dominant layer if needed)
  • audio-specific ops such as STFT / ISTFT
    • present in PyTorch audio tooling, but often missing in inference engines

Building blocks for Rust-side math/ML primitives

When full frameworks don’t fit, they use Rust tensor/matrix libraries:

  • ndarray
    • dynamic shapes and multiple backends
    • may allocate unless carefully designed
  • nalgebra
    • 2D matrices; static-shape friendly
    • “no_std compatible” angle
  • gemm / microJem / bindings
    • GEMM speed focus (homomorphic encryption startup mentioned)
  • Notes on microcontroller inference:
    • feasible, but depends heavily on matrix dimensionality and tensor representation overhead.

GPU programming outlook in Rust (future portability themes)

Key GPU deployment issues:

  • GPU programming is complex (shaders, vendor toolchains)
  • hardware lock-in risk (e.g., NVIDIA/CUDA dominance)

Projects discussed toward more portable Rust GPU programming:

  • Rust GPU
    • compiling Rust to shader code (targeting SPIR-V/Vulkan concepts)
    • direction includes compiling toward a GPU standard library
  • Rust CUDA
    • being reworked/migrated toward Rust GPU
  • WGPU + WGSL
    • used widely in UI stacks; available on WebGPU/Vulkan/Metal
  • Burn’s GPU compute backend
    • patterns like compute kernels (cube macro concept)

Missing piece highlighted:

  • DSPs and NPUs remain difficult because vendors must provide toolchains/APIs, and Rust lacks mature bindings.

Custom engine success story (real-world deployment win)

  • Stephan’s team built a custom Rust inference engine (“air 10” / “air10” as transcribed).
  • Achieved concrete improvements vs ONNX Runtime:
    • Memory: ~0.75 MB vs 18 MB (figures referenced; transcription uncertain)
    • Worst-case execution time: ~8 ms vs 16 ms
    • RAM usage reduction mentioned (phrased as “instead of 30”)
  • Design approach:
    • “dump Rust code from scratch,” minimal manual SIMD, relied on auto-vectorization
    • used a memory arena concept
    • emphasized standard/no-ownership patterns to reduce overhead
  • Claim: Rust enabled faster deployment via cross-compilation
    • “cargo build architecture → send binary → clients release quickly”

Key review / guide / tutorial takeaways

  • Framework selection guide based on:
    • model format
    • operator support
    • quantization support
    • target backend
  • Rust deployment emphasis:
    • Use wrappers like art when you need ONNX maturity and speed
    • Use burn/candle when you want Rust-native pipelines and specific formats:
      • candle: especially for GGUF
      • burn: broader Rust flexibility
    • Use tract for CPU-only deployments with strong inspection tooling
  • Performance testing and profiling:
    • recommends CLI benchmarking/profiling workflows (notably tract) to identify bottleneck layers
  • Real-time deployment advice:
    • consider allocation-free forward paths and audio-specific operators (STFT/ISTFT) for voice enhancement pipelines

Main speakers / sources

  • Stephan Eckes (main speaker; talk title credits “From Tiny Chips to Server Racks… Deploying AI Everywhere with Rust”)
  • Moderator/questioners referenced in subtitles:
    • Stefan X (appears to be the same or a subtitle misread; talk credited to Stephan Eckes)
    • David (asks a question near the end)
  • Mentioned “source” entities for technology references:
    • Meta (ExecuTorch / executorch)
    • Microsoft (ONNX)
    • Hugging Face (SafeTensors; candle ecosystem)
    • Intel (OpenVINO)
    • Sonos (acquisition/origin of tract described)
    • AMD/NVIDIA/Apple/Qualcomm (hardware backend ecosystem referenced)
    • Rust GPU / WGPU / Burn / Candle / Art / Tract (framework/project sources)

Original video