Video summary

Ornith 9B Benchmarked vs Qwen 9B - Local 8GB GPU LLM options

Main summary

Key takeaways

Technology

Overview / Purpose

Luke’s Dev Lab compares two ~9B-class local LLM options intended for 8GB VRAM GPUs, emphasizing practical performance and coding ability.

Models tested

  • Ornith 9B (GGUF, 4-bit Q4KM), sourced from Deep Reinforce AI
  • Qwen 9B (also 4-bit Q4KM), compared against a 3.5 / previous-generation Qwen baseline referenced in subtitles as 3.59B / 3.5

Both models are benchmarked using Luke’s suite plus two small coding/tutor-style challenges.

Tested Features & Methodology (Benchmarks)

Luke runs the models through a standard suite, measuring:

  1. Performance

    • Prefill speed: how fast the model ingests the prompt/context.
    • Decode speed: how fast it generates output tokens.

  2. Memory / “Needle in a Haystack”

    • Inserts a target into a large context window at depths of 0/25/50/75/100%.
    • Repeats runs 3 times per depth.
    • Tested with 256k context, then repeated with 128k and 64k when results were poor at 256k.

  3. Agency Benchmark (Tool-Use Sandbox)

    • A simulated “company” environment with tools plus decoy tools.
    • Evaluates whether the model can:
      • pick correct tools,
      • chain tools properly,
      • use a simulated internal clock.

  4. OpenAI HumanEval

    • 164 handwritten Python coding challenges.

  5. Two Coding Challenges (via a Coding Harness)

    • Expense tracker: CRUD-style UI + behavior checks
    • Memory match game: frontend/game behavior and UI state correctness

Key Results

1) Performance (Prefill / Decode)

  • Ornith vs Qwen are very close.
  • Decode speed: almost identical across settings (about mid/high-30s tokens/sec in subtitles).
  • Prefill: small differences (e.g., ~10 tokens mentioned), with both showing expected gradual slowdowns as context increases.

Performance conclusion: Comparable real-time throughput for 8GB-friendly quantized 4-bit variants.

2) Memory (“Needle in a Haystack”) — Poor for Both

  • Both models perform very badly at retrieving inserted data from large contexts.
  • Notable pattern:
    • better at start/end
    • worse in the middle (classic long-context retrieval difficulty)
  • Example behavior:
    • found the item only 1/3 at the beginning
    • found the item 3/3 at the end
  • Context performance:
    • 256k: still poor
    • 128k / 64k: only slightly better, still “not great”

Memory conclusion: These 9B models (in these configurations) struggle with large-context recall, which Luke flags as concerning.

3) Agency Benchmark (Tool Use)

  • Ornith wins overall:
    • 91% score (passed 20/22 scenarios)
    • Failures explained as:
      • one scenario using the wrong date (off by one day)
      • the “unlucky 13” scenario: larger models “can’t do,” implying higher difficulty / multi-step chaining
  • Qwen did slightly better than expected, but issues are attributed to tool-call syntax/formatting leakage:
    • it may understand what to do, but tool calls aren’t made with correct syntax, causing failures
    • Luke suggests this could matter more later for file read/write style tasks
    • for now, Ornith appears more reliable in tool calling

4) OpenAI HumanEval (Python)

  • Ornith
    • missed only two problems
    • 91% pass rate (subtitles: 148/164 passed)
    • uses an 8,000 token budget with a prompt at 4,000 tokens to push toward an answer
  • Qwen
    • answered all questions, but with lower correctness
    • 87% pass rate (subtitles: 143/164 passed)

HumanEval conclusion: For Python coding correctness at this size, Ornith looks stronger.

Coding Challenge Walkthroughs (Hands-On)

A) Expense Tracker (CRUD App)

Ornith

  • Quickly produces an app and reaches a working end state with a couple prompt iterations.
  • Observed issues:
    • initial UI behavior problems (e.g., dropdown/options not populating)
    • after receiving an error and being re-prompted, it fixes the error and dropdown options
    • some missing UX (e.g., no overall summary); category list may be incomplete and may need extra prompting

Qwen

  • UI appears more polished initially (summary/overview present; dark mode not working; nicer design).
  • Breaks repeatedly when submitting entries:
    • category dropdown “breaks” with no clear console error at first
    • requires more debugging logs/instructions
  • Luke concludes this is a clear win for Ornith because Qwen took too long / kept failing.

B) Memory Match Game

Ornith

  • Works reliably, but with initial UI/behavior issues:
    • initially reveals all tiles (not ideal)
    • then tiles become hidden but click feedback/visual indication is missing
    • needed another round to correctly enable flipping/matching
  • End state: completes correctly (“You won… matched all pairs”).

Qwen

  • Produces a nicer visual style with animations (card flip behavior).
  • Observed issues:
    • some matched-state rendering/visibility problems (cards flip, but matched behavior unclear)
    • “won” state appears incorrectly; completion messaging placement looks off
  • After further prompts/reloads, Luke says it eventually reaches the end.

Coding section conclusion: Close overall, but

  • Ornith tends to achieve correctness/functioning faster
  • Qwen tends to produce more attractive UI/design, but with more frequent functional/state bugs

Final Takeaway (Luke’s Conclusion)

  • Performance: essentially similar between the two 9B models under these 4-bit settings.
  • Memory retrieval (large context): both poor, suggesting these specific 9B models aren’t great for large-context “find the needle” tasks.
  • Agency / tool-use: Ornith is more reliable; Qwen’s weakness is often tool-call syntax/formatting, not necessarily understanding.
  • Python coding (HumanEval): Ornith performs better (91% vs 87%).
  • Small coding apps: mixed—Ornith often reaches working results quicker; Qwen sometimes has better UI but more failure/rework.

Main Speakers / Sources

  • Speaker: Luke (host of Luke’s Dev Lab)
  • Models / sources mentioned:
    • Deep Reinforce AI (hosting the Ornith 9B GGUF)
    • Qwen (Qwen 9B variant; compared against a 3.5 / previous-generation baseline)
  • Benchmark dataset/provider: OpenAI HumanEval (164 Python challenges)
  • Benchmark/testing harness: Luke’s own benchmarking test suite + “Py” (coding harness) used in the expense tracker / memory match walkthroughs

Original video