I Tested Every Best Local AI For GPU (8GB TO 24GB VRAM)

Summary

The video argues that the model file you download can matter as much as—or more than—the model name. Tier lists based only on VRAM can be misleading: a dense model that spills into system RAM may become extremely slow, while a mixture-of-experts (MoE) model can sometimes run well with most of its weights in RAM.

Key technical points

• VRAM fit is critical for dense models. Model generation repeatedly reads model weights. If some weights spill from GPU memory into system RAM, generation can be constrained by much slower RAM bandwidth. The speaker contrasts roughly 936 GB/s for an RTX 3090 with about 51 GB/s for dual-channel DDR4.
• MoE models behave differently. Qwen 3.6 35B A3B has 35 billion total parameters but uses about 3 billion per generated token. The video says this lets it run from system RAM more effectively than a dense model of similar file size.
• Quantization is not the whole story. Two files at the same nominal bit rate and similar size can perform very differently, depending on how precision is allocated across model layers. The video highlights ISTA’s quantization builds as examples.
• Tokens per second do not equal time to finish a task. Reasoning models may produce many internal “thinking” tokens before answering, so a seemingly fast model can still take a long time to complete a task.
• Context cache also uses memory. The speaker recommends prioritizing model size over cache precision for the discussed Qwen model, but cautions that this advice does not apply to every model family.

Recommendations by GPU memory

• 8 GB: Check system RAM first. With 32 GB of RAM, the video recommends Qwen 3.6 35B A3B with experts kept mostly in system memory; a reported RTX 2060 setup achieved 23 tokens per second. Without enough RAM, Gemma 4 12B at Q3 is suggested for chat and lighter tasks, but not demanding agent workflows.
• 12 GB: Use an ISTA quantized build of Qwen 3.8 27B: the 3-bit version if there is enough room for context, or 2-bit if not. The video reports that one ISTA 2-bit file outperformed a slightly larger file of the same model on bug-fixing and math tests. It advises checking who produced a file, not just its size or bit rate.
• 16 GB: Avoid the default Q4 file for Qwen 3.8 27B on Windows. The video says it leaves too little headroom for the desktop, causing silent spill into system RAM. It recommends ISTA’s 3.5-bit build instead and suggests reducing context-cache precision before shrinking the model file.
• 24 GB: Qwen 3.8 27B at Q4 is presented as a good fit with long context. The video also discusses NVIDIA Nemotron 3.5 Bolt 74B, a 2-bit model with 9 billion active parameters. Its model and vision module together use about 23.75 GB, leaving little practical room for the desktop. The speaker recommends waiting for more real-world testing before switching to it.

Other findings and cautions

• The video cites a Liquid AI LFM 2.5 24B build reaching 87 tokens per second on an RTX 3060, but reports weaker results on the bug-fixing task discussed. It notes that Liquid positions the model for tool calls rather than code generation.
• A reported RTX 3090 Founders Edition reached 112°C on its memory during heavy generation. Lowering power and increasing fan speed reportedly improved temperatures but reduced performance.
• The speaker suggests testing local models on prompts from your own work and comparing their outputs with hosted models, rather than relying only on benchmark scores.

Product mention

The sponsored segment promotes ChatLLM by Abacus AI, described as a service for comparing local-model outputs with hosted models in one interface. The speaker also says it supports image and video generation, slide decks, and app building and deployment. The stated price is $10 per month.

Main speaker and sources

• Main speaker: Kai, presenting the comparisons and recommendations.
• Referenced sources: Local Llama community reports, ISTA’s model and benchmark materials, NVIDIA’s Nemotron information, Liquid AI’s model positioning, and user reports about performance and hardware behavior.
• Sponsor: ChatLLM by Abacus AI.