Let's Run the FULL Kimi K3.1 Pro LOCALLY... 💀

Summary

The presenter tests running the full Kimi K3.1 Pro model locally on an M3 Ultra Mac Studio, using an expert-streaming setup rather than loading the model entirely into memory. The video explores how slowly a very large model can run on consumer hardware and how software optimizations may improve performance.

Setup and performance

• The full model is described as requiring roughly 1.5 TB of data or memory, making ordinary local loading impractical on the test system.
• A conventional layer-by-layer streaming approach is said to take about 30 seconds per token. The presenter instead uses expert streaming, which loads model experts as needed.
• Early results vary by configuration: one run reaches around 0.5 tokens per second, while longer generations run at roughly 0.35–0.4 tokens per second. A 10,000-token 3D solar-system generation takes about eight hours.
• The setup uses dynamic memory management: it keeps more experts in memory when space is available and fewer when other programs are using memory.
• The presenter also uses heuristics to prioritize likely-needed experts. This raises reported throughput from about 0.37 to 1.17–1.5 tokens per second—roughly three times faster. A later run reaches 2.3 tokens per second, though its coding ability has not yet been tested.

Model behavior and coding tests

• The presenter asks the model to generate interactive 3D content in HTML, including a cube and a solar system. The cube renders successfully, and the solar-system code runs without compilation errors.
• The model offers several “thinking” levels: low, middle, high, and maximum. More extensive reasoning can take substantially longer, so the presenter avoids testing maximum mode because of the expected runtime.
• Results are not perfectly deterministic: repeated cube generations differ in details such as color and background effects. The presenter attributes this partly to numerical precision and GPU computation.

Optimization and model options

• Kimi K3.1 is described as activating 16 experts per token. Loading many experts contributes to the streaming slowdown.
• The presenter discusses pruning, or “reaping,” experts to reduce the workload, but says aggressive pruning can damage the model’s capabilities. The approach used in the video prioritizes experts through heuristics rather than substantially reducing the model.
• The video mentions quantized alternatives that require around 800 GB, as well as the full streaming version. These are presented as more practical options than loading the full model uncompressed.
• The presenter says streaming and heuristics may also help run other large models on older systems, but presents this as a possibility rather than a demonstrated result for Kimi K3.1.

Availability and other platforms

• The presenter points viewers to ModelScope as another place to find models unavailable on Hugging Face, and says ModelScope offers free hosting.
• A Windows version is said to have been approved in the Microsoft Store, with a beta release under consideration.
• The presenter suggests that Nvidia systems with large amounts of system RAM may benefit from streaming weights from RAM to VRAM, but provides no benchmark results for those systems.

Guides, reviews, and tutorials

This is primarily a live performance test and technical demonstration, not a step-by-step installation tutorial. The review covers throughput, generation time, sample HTML outputs, memory management, expert-routing heuristics, and possible quantized alternatives.

Main speaker/source: The video’s presenter from the xCreate channel.