M5 Ultra vs 2 DGX Sparks… The Number You're Not Looking At

Tech comparison: M5 Ultra Max Studio (single) vs 2x DGX Sparks (cluster)

Hardware setups

• M5 Ultra Max Studio

  • “Top of the line” configuration with 256 GB memory in one box.
  • A demo unit is mentioned with an 8 TB drive.

• 2x DGX Sparks

  • Each unit has 128 GB memory → 256 GB total across the two nodes.
  • Nodes are connected by a 200 GB “fat” cable using RDMA over Converged Ethernet (RoCE), enabling remote direct memory access so nodes can write into each other’s memory directly.
  • Storage: 4 TB each → 8 TB total.
  • The cable is described as an extra cost on the Sparks side.

Key performance test: “tokens per second” (prefill vs decode)

The video measures token generation performance as:

• Closest/initial benchmark
  • DeepSeek: ~38.7 tok/s on M5 Ultra
  • vs 34.3 tok/s on dual DGX Sparks
  • Framed as “basically neck and neck.”

Two-stage inference explained

1. Prefill / prompt processing

   • Reads the whole prompt “at once”
   • Compute-heavy (GPU math)

2. Decode / token generation

   • Produces tokens one-by-one
   • Memory-bandwidth dependent (weights accessed repeatedly each token)

Why dual Sparks aren’t just “one bigger machine”

With VLM (NVIDIA’s high-throughput inference/serving engine), the model is split using tensor parallelism:

• Each model layer is cut in half across the two Sparks
• For each token:
  • both nodes compute their half,
  • then swap results over the RDMA cable before continuing the next portion.
• The cable adds back-and-forth overhead, but it allows models that may not fit on one Spark.

Model fit / quantization details

Example model: DeepSeek V4 Flash (~150–160 GB)

• Doesn’t fit in one 128 GB Spark
• Works when split across two Sparks via parallelism

Example quantized footprint:

• Each Spark holds a little over 100+ GB
• Example shown: ~111 GB per Spark

Quantization note:

• The Mac and Sparks use the “same model,” but not the same file
• Each system uses a 4-bit quantized version for smaller/faster inference.

Software/engine choice matters (Llama.cpp vs MLX)

On the Mac, switching inference engines can change speed significantly:

• DeepSeek on Llama.cpp: ~40 tok/s
• DeepSeek on MLX: ~53 tok/s
• About ~34% faster due to engine choice alone

The video emphasizes that:

• Results vary across workloads
• There may be no universally “best engine”—the best choice depends on the task.

Prompt size impact (major finding)

The biggest differentiator argued in the video: how each system handles long prompts.

Short prompt (≈2,000 tokens)

• Often favors Sparks for faster initial response
• Example:
  • DeepSeek prompt: ~2.5s on Mac vs 1.25s on Sparks

Longer prompts

As prompt size increases, the balance can shift:

• Example around 32,000 prompt tokens (“codebase race”):

  • Sparks begin streaming earlier (Mac still processing the prompt)
  • Approx stated outcome: ~17s Sparks vs ~50s Mac

• Example chart: “Quen” at ~8,000 tokens

  • Sparks significantly faster (example: ~11.2s Sparks vs 26s Ultra)

Very large prompts

• Sparks were tested up to ~128,000 tokens
• Mac was not shown at the same size
• Sparks still finished faster in the example:
  • ~72s on Sparks

Output length impact

The video also notes the opposite can happen depending on model + engine:

• Big input favors Sparks
• Big output favors Mac, at least for Quen
  • Mac’s token generation can improve over time during decoding

It also mentions an idea of disaggregated prefill/decode:

• Sparks handle prefill
• Mac handles decode
• Potentially improving end-to-end latency (referenced as a separate prototype/video)

Real-world “session” behavior: caching reduces the pain

Demos can be misleading because prompts may not be reprocessed from scratch each time.

The video mentions context caching (example: ~16,000 tokens of cache):

• First request:
  • can be much slower on Mac than Sparks
• Follow-up questions:
  • become dramatically faster because cached prompt tokens are already processed

Example stated:

• First query: ~21s Mac vs ~8.3s Sparks
• Follow-up: ~3s Mac vs ~1.4s Sparks

Concurrency / multi-user throughput and latency

The video cautions that “tokens per second” may be:

• total across all users, not per-user performance

Observations as users increase:

• Mac

  • Reading overhead grows
  • Performance drops after ~4 users
  • More time spent reading than generating

• Sparks

  • Remains strong as concurrency increases (examples cited):
    • DeepSeek peak example:
      • ~66 tok/s total at 4 users
      • drops at 8 users (example: ~46 tok/s on Mac cited)
    • Sparks continue climbing in the tested range
      • example: ~70 tok/s at 8 users

Latency pain (time to first word):

• Mac: can be over a minute in one shared scenario
• Sparks: shorter than Mac, but still slow in shared setups

Recommendation:

• Avoid very large models for many concurrent users on local setups
• Consider smaller models or other partitioning strategies.

---

Deployment complexity vs “zero setup” convenience

DGX Sparks complexity

DGX Sparks requires careful alignment:

• Match OS, kernel, drivers, firmware
• Configure VLM, multi-node operation, and many environment variables
  • “Nickel and Rocky” referenced as part of the stack
• Verify RDMA traffic
• Result: a cloud-style stack (CUDA + VLM), but with substantial setup effort

Mac convenience

• Easier to run on the Mac:
  • Llama.cpp / MLX / OMLX mentioned as simpler installs
• Also serves as a general workstation
  • e.g., rendering video and everyday development work

---

Power / thermals (load test figures)

Under heavy load, both setups run hot and noisy.

Example quoted figures:

• M5 Ultra: ~434 watts
• Spark cluster: ~410 watts

Temperature examples:

• Mac Studio hotspots around 49–50°C
• Sparks hotspot around ~48°C
• Back grill temperatures also mentioned as higher on the Mac at one point

---

Practical guidance given in the video

• Don’t assume “256 GB across two boxes” equals the same experience as one 256 GB system.
• Decision depends on your workload:

If prompts are big / long context

• Sparks can win hard on:
  • first-token latency
  • prompt processing

If you’re closer to single-user work with shorter prompts

• Mac may be simpler and competitive

• Especially with the right inference engine

• Sparks are framed as better for a small team (around ~4 users)

• And “pushing it” at ~8 users

Resources mentioned:

• NVIDIA dev site: build.nvidia.com/spark
  • Includes recipes for multi-node / VLM setups

---

Main speakers / sources

Primary speaker

• The video’s narrator/reviewer (solo presenter; no explicit name provided in the subtitles)

Referenced external sources

• Merlin AI (sponsor/featured tool for comparing models and generating web/deep research summaries)
• NVIDIA VLM and NVIDIA Spark developer site: build.nvidia.com/spark
• Ash Hart (mentioned as working on disaggregated prefill/decode ideas; referenced via Twitter)
• Apple (M5 Ultra specs) for stated memory bandwidth (quoted)