Summary of "Apple didn't have to go this hard..."

Summary of Video: “Apple didn’t have to go this hard…”

The video explores Apple’s new Mac Studio cluster featuring RDMA (Remote Direct Memory Access) over Thunderbolt, introduced in Mac OS 26.2. It focuses on the cluster’s impact on AI workloads and high-performance computing (HPC).

Key Technological Concepts and Features

RDMA over Thunderbolt

Enables multiple Macs in a cluster to share a unified memory pool.
Significantly speeds up large AI model inference by treating RAM across devices as one large resource.

Mac Studio Cluster Setup

Consists of four Mac Studios, costing nearly $40,000 total.
Two Macs with 512 GB unified memory and dozens of CPU cores cost about $10,000 each; the other two with half the RAM cost $8,000 each.
The cluster runs quietly and efficiently (under 250 watts total).
Uses Thunderbolt ports for high-speed networking instead of traditional Ethernet.
Power supply is internal, avoiding bulky external adapters, but requires Apple’s proprietary power cables.

Networking and Cabling Challenges

Thunderbolt cables lack retention mechanisms, risking accidental disconnections.
No Thunderbolt switches exist to route traffic between multiple Macs, requiring direct connections between all devices.
This limits cluster size to four Macs for RDMA.
Ethernet is 10 GbE but slower and less efficient compared to Thunderbolt RDMA.

Comparison with Other AI Systems

Benchmarked against Nvidia DGX Spark (Dell Pro Max) and AMD AI Max Plus 395 systems.
Apple’s M3 Ultra Mac Studio outperforms competitors in Geekbench and FP64 HPL benchmarks, breaking a teraflop on a single node.
Superior power efficiency with idle power under 10 watts.
Handles large AI models (e.g., Llama 70B, DeepSeek R1, Kimmy K2) better than competitors, sometimes requiring fewer nodes.

Software and Management

Mac OS cluster management is more cumbersome than Linux; lacks full system upgrade over SSH, requiring manual UI interaction or screen sharing.
Automations via shell scripts and Ansible are possible but less streamlined.
RDMA must be enabled manually via recovery mode terminal commands on each Mac.
Tested AI workloads with Exo (an open-source AI clustering tool) and llama.cpp.
Exo leverages RDMA for pooling memory and speeding up inference, while llama.cpp’s RPC clustering slows down with more nodes.
RDMA over Thunderbolt is still early-stage tech with bugs; e.g., HPL tests over Thunderbolt cause system crashes.

Exo Clustering Tool

Open-source, Apache 2.0 licensed, but development has been somewhat secretive, possibly due to Apple collaboration.
Shows promise for scaling AI inference across Macs with RDMA.
Future plans include integrating Nvidia DGX Spark for prompt processing and possibly Raspberry Pi support.

Potential Improvements and Future Questions

Desire for Apple to adopt QSFP networking ports (like Nvidia and AMD systems) for better retention and scalability.
Suggestion for Apple to upgrade Mac Pro with PCIe expansion and 100 Gb QSFP networking for research labs.
Possibility of SMB Direct support over Thunderbolt RDMA for seamless network file sharing, beneficial for video editing and other workflows.
Questions about future Apple silicon iterations like an M5 Ultra.

Overall Impressions

Mac Studios are powerful, efficient, and quiet workstations that excel in AI and scientific computing tasks.
Despite higher cost, they outperform similarly priced AI desktop systems in performance and efficiency.
Managing a Mac cluster is more complex than Linux-based clusters, and Thunderbolt networking limits scalability.
The cluster and Exo tool represent a promising but early-stage approach to local AI compute clusters.

Reviews, Guides, and Tutorials

Hands-on testing and benchmarking of Mac Studio cluster vs. Nvidia and AMD AI systems.
Step-by-step enabling of RDMA on Mac Studios via recovery mode terminal commands.
Performance comparison of AI models (Llama 3B, Llama 70B, DeepSeek R1, Kimmy K2) across single and clustered Macs.
Discussion of cluster management challenges and automation tips using Ansible.
Practical insights into cabling and rack-mounting Mac Studios for cluster setups.