Video summary

Qwen3.8 27B: Same Model, Three Harnesses, One Clear Winner

Main summary

Key takeaways

Technology

Summary of the video (tech concepts, harness features, test results)

The video compares three coding “harnesses” used to drive Qwen3.8 27B for practical coding tasks. A harness is described as a program (similar in spirit to CodeX/Claude Code) that wraps the model and typically enables tool-calling and iterative generation to improve results.

Harnesses tested

  1. Pi Harness / Pi Coding Agent

    • Runs like a terminal/CLI tool; the user launches it inside a local projects folder.
    • Configured via a local settings file (models.json) and requires an API key (engine mentioned as “Unslaw Studio”).
    • Supports thinking/reasoning level controls: minimal/low/medium/high/extra high.
    • The creator notes the harness likely maps/manipulates Qwen’s native reasoning levels (Qwen has “low/medium/X high”) into its own set.
    • The creator tests the same Minecraft prompt multiple times using low, medium, and X high.

  2. Hermes Agent (connected via Discord)

    • Uses Discord chat to interact with a “Hermes agent” for code generation / tool use.
    • The creator ensures all profiles/agents/vision models use Qwen3.8 27B to avoid falling back to other models.

  3. DeepSeek Harness (browser-based, looks like CodeX)

    • Appears to be hosted locally (mentioned localhost:3001).
    • Required restarting/starting the server after the computer restarts.
    • Emphasized as producing some of the best “oneshot” results for the creator’s large Minecraft prompt.

Test 1: Minecraft clone generation (2-part evaluation via iterations)

Common evaluation prompt

A very long ~10-page Minecraft prompt is used. The creator describes it as unusually demanding; only Fable 5 with Claude is said to have “oneshot” it in prior experience.

Performance comparison (Pi Harness)

  • Pi + low thinking level: winner within Pi

    • Eventually reaches a playable-looking Minecraft environment after multiple back-and-forth iterations.
    • Major issues noted:
      • Textures missing on block sides at first; needed prompts/screenshots to get fixed.
      • Crafting system does not work (crafting table/steps fail).
      • Performance stutters / frame rate swings widely (e.g., ~240 down to ~30 FPS).
    • Achievements/features observed as working:
      • Block breaking animations and behavior.
      • Zombies/creepers present with animations; they can damage the player.
      • Improved water escape behavior (jumping out of water more like expected behavior).

  • Pi + medium and Pi + X high: failures

    • Medium: “alien looking” entities; textures not generated; fails visually.
    • X high: worse gameplay controls; WSAD movement reversed (A/D and W/S behavior wrong).

Performance comparison (Hermes Agent)

  • Hermes ranks third overall for Minecraft.
  • The creator tried multiple revisions:
    • Early generations were “bad” and hard to make playable.
    • Even in the final version:
      • Breaking animations are missing or nearly invisible.
      • Crafting is only partially functional (crafting table isn’t correct; craft flow doesn’t fully match Minecraft expectations).
      • Night cycle exists; hostile mobs are inconsistent:
        • Zombies may be present but attacks/hit feedback can be missing.
      • Biomes/world feel less diverse; terrain has problematic density/paths where you can’t walk through at points.
    • Humans appeared even though not requested (suggests the harness/model introduced web-like content or unwanted assumptions).

Performance comparison (DeepSeek Harness)

  • DeepSeek is the best for the Minecraft test.
  • Notes:
    • Achieved the best oneshot result from a “local model” the creator has seen for this specific prompt.
    • Works on low; medium failed completely (creator states low thinking level is best for this model).
  • Improvements and “final output”:
    • Iterations addressed missed details from the original prompt (e.g., verifying the world is theoretically infinite, plus minor bug fixes).
    • Strong gameplay functionality:
      • Block holding animation (selected block appears in-hand).
      • Floating blocks behavior resembling Minecraft visuals.
      • Crafting system works end-to-end (wood breaks faster after fixes; crafting steps like sticks → pickaxe work).
      • Hostiles exist: zombies and creepers, including creeper explosions.
    • Remaining issues:
      • Water physics / water fill not fully correct (water doesn’t fill all the way; player may need to break blocks to exit water depending on conditions).
    • World/seed behavior:
      • Appears to spawn you in a random spot, and mentions seed generation.

Minecraft verdict: DeepSeek wins overall.

Test 2: Color palette generator website generation

Shared website prompt

A ~17-page prompt describing a color palette generator web app is used. The prompt was derived from a ChatGPT prompt for CodeX; the creator hosts prompts for viewers.

For fairness, the creator asks each harness to:

  • implement all suggested improvements it recommended.

Pi Harness website results

  • Produced a functional UI with:
    • modes like random, pastel, vibrant, dark, light
    • ability to change number of colors (up to eight)
    • image input capability
    • exports described as CSS variables / Tailwind (per prompt requirements)
  • Visual quality:
    • Works well, but the creator feels it’s not the best-looking among the three.

Hermes Agent website results

  • Took longest to generate (reason unclear).
  • Produced a “pretty decent” site with good styling:
    • palette generation (random/pastel/vibrant/dark) appears correct
    • save palette and export features
    • FAQ/about/instructions
    • professional styling (creator “likes the style” especially the dark theme)
  • Visual score:
    • Hermes is given an edge over Pi based on appearance for this test.

DeepSeek Harness website results

  • UI looks the most professional and closest to an existing creator-made benchmark site.
  • Features working/mentioned:
    • generate palettes correctly (pastel/vibrant/complementary tested)
    • export palette in CSS and multiple formats
    • save palette and download SVG
    • likely supports multiple styling modes; one run included image upload but this specific run may not show that
  • Visual verdict:
    • DeepSeek is considered the best-looking overall.

Website verdict: DeepSeek ranks #1 for style/professional polish.

Overall final ranking (as stated by the creator)

  1. DeepSeek Harness — #1
  2. Pi Harness — #2
  3. Hermes Agent — #3

Main speakers / sources

  • Main speaker: the YouTube creator narrating and running the harness tests (no other named speakers in the subtitles).
  • Sources referenced (not speakers):
    • Qwen3.8 27B model
    • Harness tools: Pi Coding Agent, Hermes Agent, DeepSeek Harness
    • Mentioned prior model/harnesses: Claude Code, CodeX
    • Referenced third-party engine: “Unslaw Studio” (as the API engine used by the creator)

Original video