I Run Full DeepSeek v4.1 Flash on a 12GB GPU

Summary

The video examines whether the full DeepSeek V4.1 Flash can run on a 12GB graphics card. It argues that “full versus fake” is less useful than asking how many bytes the computer must move for each generated token.

What “full model” means

DeepSeek’s released model is already mixed-precision: its dense weights are FP8, while its expert weights—the bulk of the model—are FP4. The narrator estimates the complete release at about 510GB, rather than the roughly 1.5TB an uncompressed 16-bit model would require. A two-bit version therefore cuts the experts’ precision in half; it does not reduce it to one-eighth of an imagined 16-bit original.

The model has 40 layers with 384 experts per layer. It selects six experts in each layer for a token, so 240 experts contribute to each generated token. The narrator estimates the following data movement:

• Original FP4 experts: about 4.5GB per token
• Promethevs two-bit copy: about 2.8GB per token
• Strata two-bit files: about 2.1GB per token, or around 1.8GB when using five experts

A 12GB GPU cannot hold the model. Instead, it keeps a small set of frequently used experts in VRAM while the rest are served from system RAM and fast NVMe storage. In the reference setup, the GPU holds about 78 experts—roughly 0.9GB, or half a percent of the model.

Speed, accuracy, and the Promethevs approach

On a Ryzen 9 3950X system with 64GB RAM, an RTX 3060 12GB, and two NVMe drives, Promethevs reportedly ran the original FP4 weights at 2.14 tokens per second—approximately one token every half-second. A separate user report cited about 1.6 tokens per second on an RTX 5060 with 8GB VRAM, rising to 2.4 with a RAM cache.

Strata’s smaller two-bit files can be faster because less data must be read. The video cites about 4.7 tokens per second on an RTX 5070 in one setup. Other independent results were less consistent: AI Receipts measured 1.84 tokens per second on a 5070, while No Place Like Localhost reported 2.8 on a 4070 and 4.7 after adding a 24GB GPU. One coding task in that test took 11 minutes and 30 seconds.

Promethevs aims to improve speed and recover some accuracy through two techniques:

• Compress then correct: It generates from the two-bit copy, then uses stored correction data to recompute uncertain tokens with the original FP4 weights. Its audit found that the two-bit copy matched the full model’s next-token choice 97.4% of the time across 960 words; corrections raised this to 98.33%. Corrections add substantial time when triggered.
• Draft-head decoding: The model includes a draft head that proposes several words, which the engine checks together in one pass. In one run, it accepted 122 of 129 guesses, averaging 4.13 words committed per round. This helps explain Promethevs’ reported rate of around 7 tokens per second.

The narrator cautions that matching the next token is not the same as reproducing the same complete answer: an early difference changes the context for every later token. In one outside check, outputs diverged after only a few words. Strata’s reported next-token agreement was lower—83.7% for its two-bit file and 87.2% for a 2.25-bit version—and its two-bit perplexity rose 15.5%. An AI Receipts blind comparison favored the cloud answer over the local one, but it was a single test and should not be treated as conclusive.

Hardware tuning and prompt processing

A reported RTX 4090 system initially managed 2.66 tokens per second on code and 1.20 on reasoning. The engine had left only 384MB of VRAM free while the GPU also drove the display. Reserving 3GB for graphics memory reportedly raised results to 9.75 tokens per second on code and 5.23 on reasoning, while reducing correction time. The video recommends setting this reserve before benchmarking a GPU that also drives the desktop.

Generation speed is not the whole story. On a 1,163-token prompt, Promethevs reportedly took about 126 seconds to produce the first word, though follow-up messages in the same conversation could start in around two seconds. Strata was reported to prefill much faster—up to 231 tokens per second at an 8K context—so the narrator warns against mistaking prompt-reading speed for output speed. Strata’s default 8,192-token context would also be too small for one cited 9,100-token coding prompt.

Practical trade-offs and recommendations

Promethevs requires roughly 1.08TB of disk space, is described as Windows-only, supports one Nvidia RTX 30- or 40-series card, and took about four hours to build on the reference system. Strata needs about 257GB. The video also mentions an Unsloth lossless 508GB build for systems with 128GB RAM or more, running through llama.cpp.

The narrator compares local inference with DeepSeek’s API, citing an off-peak price of $0.60 per million output tokens. At 7.4 tokens per second running continuously for a month, Promethevs’ output would cost about $11.51 at that rate, before electricity. A $10 Open Code Go subscription is presented as a better-value option for many users, though local inference remains relevant where prompts cannot leave the organization, such as in privacy-sensitive or regulated work.

The video’s practical guidance is:

• With 32GB RAM: The narrator advises against this setup, citing slow real-world results.
• With a 12GB GPU and 64GB RAM: Try Strata’s two-bit build; enable its full-quality mode to use all six experts, increase the context window above 8K, and turn thinking mode off for quick tasks.
• With Windows, an RTX 30/40 GPU, and about 1TB of free storage: Promethevs may be worth testing. Reserve at least 3GB of VRAM if the GPU also drives the display.
• With a Mac and 64GB unified memory: The video says Strata has a DeepSeek build for this setup.
• If privacy is not a barrier or the work is paid: Consider the API, which avoids the local hardware and speed costs.

The overall conclusion is that local DeepSeek inference is possible on a 12GB GPU, but it relies on streaming weights from RAM and storage. Performance depends heavily on data movement, prompt length, hardware configuration, and the chosen quantization.

Main speakers and sources

• Kai: The video’s narrator and reviewer
• Ryan: Developer of Promethevs, whose method and benchmark results are discussed
• Strata: Documentation and performance figures
• AI Receipts and No Place Like Localhost: Cited for independent tests
• Additional user reports and GitHub issue feedback cited for hardware results and tuning observations