Cómo Obtener 256GB De VRAM Para IA Local (Todas Las Formas)

Summary

The video compares realistic ways to run large AI models locally with roughly 256 GB of memory. It argues that purchase price per gigabyte can be misleading. A better measure is cost per useful year: an older, cheaper server may be expensive to operate and lose software support sooner, while a newer machine may cost more upfront but remain useful longer.

Why 256 GB matters—and how much the model actually gets

• Around 256 GB can accommodate several large open-source models in 4-bit or 6-bit quantizations. Quantization reduces memory use by storing model values at lower precision, potentially at the cost of quality.
• Mixture-of-experts (MoE) models activate only some of their parameters for each token, which can make them faster than dense models. However, their inactive experts still need to stay in memory.
• The advertised memory figure is not the same as the amount available to the model. The operating system and the conversation’s KV cache also use memory, especially with long context. The narrator recommends budgeting about 220–235 GB for the model.
• Macs commonly limit GPU memory use by default. The video describes raising that limit with sudo sysctl iogpu.wired_limit_mb and applying the setting at login so it persists after restarts. AMD systems also need memory-limit configuration; in one test, the narrator cites around 108 GB as a stable practical limit per 128 GB Strix Halo machine.
• The narrator evaluates machines using three “clocks”: software support, how long relevant models fit in memory, and ownership cost, including electricity and resale value.

Options reviewed

• Used servers with eight 32 GB Nvidia V100s: Listings cited range from about $5,623 to $8,955. They offer a very low purchase cost per gigabyte and have been shown running large models, but they are loud and heavy, require 240 V power, and can draw several kilowatts. CUDA 13 and newer tools are dropping Volta support, though older CUDA builds and community patches can still run some workloads. The narrator treats this as a short-term, roughly two-year project—not a long-term investment.
• Apple Mac Studio M5 Ultra, 256 GB: The cited price is $9,499, with a possible $10,799 configuration depending on which GPU option Apple requires. It offers high memory bandwidth, relatively low power use, quiet operation, and a comparatively strong software outlook through Apple’s MLX ecosystem. Its main performance drawback is slower prefill—processing long prompts—than some Nvidia systems. The narrator recommends checking Apple’s configurator carefully.
• Used Mac Studio M3 Ultra, 256 GB: Recent second-hand sales cited are around $7,700–$7,900. It is slower than the M5 Ultra, particularly on long prompts, but its resale-market price has remained unusually high.
• Two Nvidia DGX Spark systems: Two 128 GB units need a cable and cost about $13,900 plus the cable at the cited prices. Their CUDA ecosystem and strong prompt processing may suit programming agents that repeatedly read large codebases. In one comparison, generation speed was close to the M5 Ultra, while the Sparks handled a long prompt considerably faster. Their sharp price increases and uncertain resale value weaken the case.
• Two AMD Strix Halo mini PCs: The video estimates about $6,898 for two Framework 128 GB systems at listed prices, though availability and prices vary. Together, they can provide roughly 210–240 GB of usable memory depending on configuration, with relatively low power use. Trade-offs include modest speed, slow prefill, and the complexity of connecting and configuring two machines. The narrator also suggests comparing their price with a single newer 192 GB “Gorgon Halo” system.
• Multiple new graphics cards: Professional Nvidia cards offer much higher cost and power efficiency per card than gaming cards, but are very expensive. Four RTX Pro 5000 cards provide 288 GB; four RTX Pro 6000 cards have produced high throughput in some tests, though results vary with software and kernels. Eight RTX 5090s would consume about 4.6 kW just for the GPUs, and the narrator says current prices make them particularly unattractive. An eight-card AMD R9700 build is also mentioned, along with the complexity of assembling and troubleshooting it.
• Older AMD MI50 systems: These can be inexpensive and have community reports of respectable model speeds, but require substantial power and cooling. Like V100s, they face diminishing support in current software and increasingly depend on community-maintained builds.

Model changes, renting, and the buying decision

A central example is DeepSeek’s move from a smaller Flash model to a much larger one. The newer model can fit on a 256 GB machine only with very low-bit quantization and careful memory management; higher quantizations leave little room for conversation context. Support for it also arrived in tools at different times. The narrator uses this to illustrate the model clock: a machine’s memory capacity stays fixed while the models people want may grow.

For occasional use, the video argues that cloud rental or paying for API tokens can be cheaper than owning a machine. Local hardware can make more sense when it is used heavily, when data cannot leave the premises, when model weights must remain fixed, or when a particular local CUDA workflow is required.

Existing PCs may also be worth testing. The video cites community examples running a low-bit DeepSeek model with a 24 GB RTX 3090, substantial system RAM, and a fast NVMe drive, though performance varies widely.

The narrator’s rough cost-per-year comparison assumes five useful years for the M5 Ultra and two for an old server, and includes US residential electricity rates and eight hours of daily use. Under those assumptions, the Mac comes out cheaper per useful year despite its higher purchase price. The result is sensitive to the assumed lifespan, resale value, electricity cost, and utilization.

Practical guidance from the video

Before buying, the narrator recommends checking:

1. How much memory the model can actually use after accounting for system and KV-cache requirements.
2. The newest CUDA or ROCm version supported by the hardware, and whether the tools you rely on still compile for it.
3. The size and quantization of the model you intend to run—and whether its next version may exceed the available memory.
4. Power consumption and local electricity costs.
5. Expected useful life and resale value.

For model distribution, the video notes that tensor parallelism divides work across cards. It recommends choosing two, four, or eight cards rather than three, and says cards without NVLink may require less efficient pipeline parallelism.

Overall recommendation

The narrator favors the M5 Ultra for longevity and lower hassle, the used M3 Ultra as a cheaper Mac option, and DGX Spark for workflows that specifically benefit from CUDA or fast prompt processing. AMD mini PCs are positioned as a more hands-on budget route; professional GPUs suit heavily utilized team workloads; and old V100/MI50 servers are best for buyers with cheap power, space, and tolerance for software maintenance.

Main speaker and sources

• Main speaker: Jorge Marfil, the video’s narrator.
• Sources discussed or cited: Apple and Nvidia documentation; CUDA, ROCm, MLX, llama.cpp, vLLM, LM Studio, and Ollama documentation or release information; benchmark coverage from Tom’s Hardware, MacStories, and Mind Studio; and reports, tests, and discussion posts from users on Reddit and Hacker News.