Every Ways to Get 128GB VRAM for Local AI at Full Context

Summary

The video compares the main ways to get roughly 128 GB of memory for running large local AI models. Its central finding is that, for Qwen 3.8 Flash-Next, a 128 GB machine may mainly buy 4-bit model quality, rather than a dramatic speed increase. The model’s sparse design reads only about 6 billion of its 125 billion parameters for each generated word, so a smaller GPU can keep the active weights in VRAM and offload other data to system RAM.

Key Technical Findings

• Flash-Next’s 4-bit files are about 94–124 GB before context, which explains the appeal of 128 GB. However, advertised memory capacity can overstate what the GPU can actually use:
  • Windows systems with AMD Ryzen AI Max+ 395 (“Strix Halo”) generally allow the GPU to claim at most about 96 GB by default; Linux can reportedly expose more.
  • A 128 GB Mac also limits GPU-locked memory to around 96 GB unless the user raises the limit in Terminal.
  • The DGX Spark has about 122 GB usable in the described setup, so a 124 GB model file can spill onto the SSD.
• Inference software matters greatly. In a reported 160,000-token test on the same Strix Halo machine, two engines took about 23 minutes versus 2 minutes to read the prompt. Strata and other engines also achieve much higher speeds than some default configurations.
• A 12 GB RTX 5070 plus 64 GB of system RAM can run the 3-bit build through Strata at about 50–62 tokens per second, with roughly 1,750 tokens per second reading a 32,000-token prompt. A reported RTX 3090 setup also reaches 60–81 tokens per second on the 3-bit build.
• Moving that PC approach to 4-bit can sharply reduce generation speed. A reported laptop-GPU setup with 192 GB of RAM managed 21–33 tokens per second at 4-bit, versus 53–59 at 3.5-bit. In the video’s comparison, the 3-bit setup is competitive with 128 GB machines, while 4-bit is the more demanding step.

Hardware Options Discussed

AMD Strix Halo Mini PCs

The least expensive dedicated 128 GB route discussed, starting around $3,099 for a Bosgame system; Framework and Minisforum versions cost more. With Linux and a suitable inference engine, reported 4-bit generation is 47–54 tokens per second. A 192 GB Ryzen AI Max Pro 495 system is also described as forthcoming.

Mac Studio with M5 Max

Around $5,999 for the 128 GB configuration, with 614 GB/s memory bandwidth. MLX Serve reports roughly 40 tokens per second on prose and 75 on code. The video praises its quiet operation and software experience, but notes a lack of published prompt-reading results for this model and says the memory limit needs manual adjustment.

Nvidia DGX Spark

Listed at $6,950 after a reported price increase. It generates at about 41 tokens per second median in the cited results, with a draft-ahead feature helping performance. Its prompt-reading advantage over Strix Halo is much smaller on Flash-Next than in older comparisons. The video argues it is most compelling when CUDA is specifically required.

RTX Spark Laptops and Desktops

Microsoft’s 128 GB products are priced around $5,899–$5,999. The video cautions that Microsoft’s performance claims use pre-production hardware and nonstandard comparison conditions; it recommends waiting for independent Flash-Next benchmarks.

Multi-GPU Server Builds

A community build using four used Radeon Pro V620 cards, a used EPYC system, 256 GB RAM, and a large power supply reportedly cost about $3,000 and generated at 60–70 tokens per second. It consumes roughly 500–600 W under load, and the low-cost cards became harder to find after the build drew attention.

Other options—including MI50, Arc Pro B70, Radeon AI Pro R9700, and RTX Pro 6000—are discussed, but their costs or practical drawbacks are substantial.

Existing Gaming PCs

For someone who already has a suitable GPU, adding 64 GB of RAM and running the 3-bit model may be a much cheaper first step. The speaker estimates about $880 for the RAM-based upgrade in one scenario. Building a new PC around an RTX 5070 is estimated at about $2,400.

Buying Guidance and Limitations

The speaker recommends trying the 3-bit Strata setup on an existing gaming PC before buying a 128 GB machine; choosing Strix Halo with Linux for a relatively affordable 4-bit system; considering the Mac Studio for quiet operation and strong generation speed; and buying the DGX Spark only when CUDA is a real requirement. The speaker advises waiting for independent benchmarks of the RTX Spark and reserving multi-card servers for workloads that need to serve multiple users or agents.

The comparison has important caveats: Strata’s speed gains rely on Flash-Next’s particular architecture and may not carry over to other models. The video also notes that there is not yet a direct comparison of 3-bit and 4-bit quality in long agent sessions—the kind of workload that could justify buying more memory. For occasional use, the speaker suggests comparing hardware costs with cloud rental or hosted-model pricing.

Main Speaker and Sources

The video is presented by Kai. Its evidence includes benchmarks and reports from community users and inference-engine developers, alongside product and performance claims from AMD, Nvidia, and Microsoft.