Дорогие видеокарты для ИИ больше НЕ НУЖНЫ?

Summary

The video analyzes whether Strata can run large local language models on a mainstream gaming PC—and why performance claims such as 93 tokens per second on an RTX 5070 can coexist with results of just 4–6 tokens per second.

How Strata works

The example setup uses an RTX 5070 with 12 GB of VRAM and 64 GB of system RAM. The featured Qwen model is described as having 125 billion parameters, compressed to around 2.4 bits, with a file size of roughly 66 GB—far too large to fit entirely in the GPU’s memory.

The model uses a Mixture of Experts (MoE) architecture. Although it has 125 billion parameters overall, only about 6 billion are active for a given token. Strata uses this to divide the workload:

• VRAM keeps frequently used experts and handles GPU computations.
• System RAM holds much of the remaining model data.
• The CPU can calculate with experts that are not currently in the GPU cache, working in parallel with the GPU.
• The SSD can supply pieces of an auxiliary Ingram table through the operating system’s file cache.

Strata’s adaptive caching can change which experts stay in VRAM as usage shifts. The developers estimate that each additional gigabyte of VRAM can hold around 700 experts, but the video notes that the documentation does not provide a general cache-hit rate.

The setup is therefore not “a huge model made small,” but a way of distributing its data and calculations across the GPU, RAM, CPU, and storage. The trade-off is that RAM capacity and bandwidth matter greatly; the video estimates the model alone needs about 34 GB of RAM, making 64 GB a more practical setup once the operating system and other applications are included.

Performance techniques and benchmark differences

Strata also uses multi-token prediction: it proposes several tokens ahead, then has the main model check them. The developers claim this can speed generation by up to 1.8×.

Further optimizations batch some operations—such as routing and normalization—instead of repeating them for each predicted token.

The 93 versus 4–6 tokens-per-second results are not measurements of the same model. The faster figure is for Qwen; the slower figure is for a substantially heavier DeepSeek variant. The video says one DeepSeek version needs about 250 GB of storage, so relying on SSD data transfers can become a major bottleneck.

The comparison illustrates that GPU speed alone does not determine performance: the model, memory capacity, and whether data must be fetched from slower storage all matter.

Limitations and practical guidance

• Strata is optimized for specific model architectures, rather than being a universal accelerator for any local model.
• Longer prompts can slow generation. One cited Qwen test on an RTX 5070 reportedly saw about a 20% speed drop at 128,000 tokens. Users have also reported crashes with very long requests, though the video says some issues have been fixed and behavior depends on the engine version and settings.
• Model quality is uncertain. The compressed-model authors reported results slightly better than a higher-precision original, but the video cautions that independent testing is needed. One Hacker News user reported more than three times the position-detection error versus Llama in their image-processing test; this is a single reported result, not evidence that Strata is worse at every task.
• A user with an RTX 3080 and less RAM reportedly achieved around 30 tokens per second on a programming model. Actual results will depend on the CPU, memory speed, model, and configuration.
• The video suggests considering Strata if you already have hardware close to the RTX 5070/64 GB RAM setup and want to experiment with local Qwen models, coding, or AI agents. Before buying, compare the cost of a full RAM upgrade—not just the GPU. For very long contexts, image accuracy, or unsupported models, the narrator suggests considering llama.cpp instead.

Guides or reviews

There is no step-by-step setup tutorial. The video is primarily a technical analysis of Strata’s architecture and benchmarks, with practical advice on evaluating hardware requirements, performance claims, and model quality before buying components.

Main speaker and sources

The main speaker is the video’s narrator, who analyzes Strata’s documentation and published developer benchmarks. The discussion also cites user reports from GitHub and Hacker News, plus an RTX 3080 user’s reported performance.