Underdog Saluki 27B tested - Local LLM setup

Summary

The video evaluates Underdog Saluki 27B from Conway Research, a heavily quantized model that the presenter says is compressed to under 8 GB. With an appropriate context setting, it can run on a 12 GB GPU and fit more comfortably on a 16 GB card. The main trade-off is reduced reliability on complex reasoning and multi-step coding tasks.

Setup and performance

• The model was run with llama.cpp on Ubuntu Server on a system with 32 GB of DDR4 RAM. The presenter describes a 16 GB GPU setup and reports benchmark results from a 70 W workstation GPU.
• Decoding speed was about 18–19 tokens per second, which is usable for interactive work. Reported prompt-processing speeds ranged from roughly 340–370 tokens per second uncached to much higher rates with caching.
• In a long-context retrieval test, the model found the target information in all 15 runs. Results were strongest near the start of the context and became somewhat less consistent deeper in, though retrieval still succeeded.

Reasoning and coding benchmarks

• Reasoning: 34/48 correct (71%). Performance was solid on easy and intermediate questions, weaker on difficult ones, and poor at expert difficulty (2/12 correct).
• HumanEval Remix: 64% overall success across 100 Python tasks. It did not answer 18 tasks; among answered tasks, the reported success rate was 78%.
• The presenter attributes much of the decline on harder tasks to the model’s aggressive quantization.

Practical software tests

• Kanban web app: A good result, with working card creation and movement, status columns, and light/dark modes. No follow-up prompts were needed. The presenter switched to the open-source OpenCode tool after encountering token-limit problems with Pi and DeepSeek harnesses.
• Sand physics: A decent simulation with flowing water, walls, and acid that dissolves materials. It needed one follow-up prompt and used 41% of the context.
• Dungeon crawler: Produced a playable dungeon with visibility and exploration mechanics, but required a correction after initially returning an empty dungeon. Some visual issues remained.
• Blender: Created a partially successful “Starfall” lantern model, but the task was ultimately judged a failure because attempts to capture screenshots through Blender caused crashes. The presenter suggests the model may still be useful for simpler Blender objects.
• Godot: Failed the more demanding game-development test. Problems included inverted controls, movement and jumping issues, missing gameplay elements, and a broken respawn flow. It took nine rounds of correction.

Overall assessment

The presenter considers Saluki 27B a reasonable option when GPU memory is limited, especially for simpler web-development tasks and smaller creative projects. It performed respectably on basic interactive tests, but struggled as tasks grew more complex or required many steps and tool interactions. The conclusion is to keep expectations realistic: an under-8-GB quantized model should not be expected to match higher-quality quantizations or handle demanding game-development workflows.

Tests covered

Performance, long-context retrieval, reasoning, HumanEval Remix, Kanban, sand physics, dungeon generation, Blender, and Godot.

Main speaker/source: Luke’s Dev Lab presenter. The model is from Conway Research.