Video summary

¿Qué es esto del Harness Engineering?

Main summary

Key takeaways

Technology

Summary of Technological Concepts & “Harness Engineering” (from the subtitles)

Problem with current AI coding/chat tools

Tools like ChatGPT, Cloud Code, and Copilot can sometimes work extremely well, but they often fail on what seem like “simple” tasks. The video argues that many improvements in AI-assisted development come less from better base models and more from creating better “environments” in which those models run.

Definition: Harnesses

A harness is the environment/framework around an AI model (agent) that controls:

  • What context is provided to the model
  • What tools/actions the agent can use
  • What the system remembers (and where)
  • How outcomes are validated (rather than blindly accepted)

Metaphor: “reins/saddle” to control a “runaway horse” (the model generating lots of code).

What the harness environment includes

A harness typically provides:

  • Context to send to the model (prompting context, instructions, and constraints)

  • Tools/actions the model can use (e.g., file exploration, command execution)

  • A separate memory system to supply relevant context about the current work and what must be remembered

  • Validation mechanisms so the system can verify results instead of trusting the model’s claims

Core principle: simpler harnesses can perform better

The video claims that overly complex harnesses—especially those giving agents hyper-specialized tools—can worsen performance.

  • It cites an article (from Oversell) where removing about 80% of tools improved:
    • Speed
    • Cost
  • It also provides a tooling change example:
    • Replace many specialized database/SQL tools with a small set of basic Unix-like commands (e.g., reading/listing/searching).
    • Reported gains: 3x+ speed increase and ~37% less token usage
    • The simplified tool setup reportedly outperformed the complex one on the requests tested.

Context degradation & memory management

Even with large context windows, the video argues that performance degrades as context grows, often before the window is full.

  • It references a GitHub issue suggesting degradation starts around 20% context usage
  • It recommends clearing context / restarting around 40%

Therefore, harnesses should:

  • Prevent context flooding
  • Use external memory (files/databases) and only extract the relevant pieces into the context window

Long coding sessions: controlling what the agent does

The video references Anthropic guidance for building effective harnesses for long sessions and introduces structured task tracking using files, such as:

  • A JSON/“Jason-like” file describing:
    • Tasks
    • Acceptance criteria
    • Status (e.g., pending/done)

This structure also supports multiple agents in parallel by letting each agent consult shared task/memory files.

Sub-agents pattern (and avoiding “inherit all context”)

A parent/orchestrator agent can spawn sub-agents for specific work, such as:

  • investigation
  • implementation

Key emphasis: sub-agents should have reduced context so they stay efficient and focused, rather than inheriting everything from the parent.

Verification: AI must prove work is correct

The video warns that AI-generated code can sound correct but still be wrong. You can’t trust the model to merely “say it’s done.”

A harness should include verification tools, such as:

  • Automated tests
  • Possibly browser-based testing (subtitles reference “Papetir” / Chrome tooling)
  • Emerging approaches may use AI as a code review/validation layer, but still within a harness-style verification workflow

Multi-agent architecture example (Anthropic)

The video cites “How We Built Our Multi-Agent Research System.”

A described structure:

  • A research/orchestration agent delegates work
  • Smaller agents:
    • search and extract information
    • store relevant results in memory

The orchestrator iteratively assigns tasks until completion, then returns results.

Three “pillars” of the harness architecture

  1. Use the repository itself as the system Don’t treat it like a generic chatbot—the repo provides context and rules.

  2. Multi-agent orchestration Leader/spawner plus worker sub-agents.

  3. Verification The harness verifies outputs and can even help self-improve configuration.

Concrete example project/tutorial (downloadable)

The video describes a starter project architecture, including:

  • Agent definitions in a folder named like .… cloud (subtitles unclear), representing “agent definitions”
  • Roles such as:
    • implementer
    • lead/orchestrator
    • reviewer
  • A central entry-point file (named like agents.m), which sets initial context/rules
  • A required initialization script: init.sh
    • scaffolds the project
    • runs quality checks/tests
    • stops the agent if tests fail or required files are missing
  • A features/tasks file (example name appears like featurist.jonjason)
    • includes features with acceptance criteria and status (done/pending)
  • Repository-provided protocols
    • Agents must read:
      • Agents MD
      • architecture docs
      • conventions
    • They shouldn’t mark tasks “done” unless tests pass

“Progress” folder as memory

Agents write outputs to a progress directory, such as:

  • exploration results
  • metrics
  • added dependencies
  • a per-run history log

This reduces the need to reload the entire repo into context (“avoid the game of telephone”), preventing messy handoffs.

Possible upgrades mentioned:

  • storing memory in a local database (e.g., SQLite/MongoDB)
  • or a remote shared DB so agents can share memory across runs

Reviewer agent behavior (code review + enforcement)

A reviewer agent:

  • reads conventions and repository memory
  • checks whether:
    • architectural requirements are respected
    • conventions are followed
    • init.sh passes (tests are green)
  • can provide feedback back to the implementer
  • subtitles suggest the reviewer/harness may also edit repo files (e.g., prompts/docs) for self-improvement

Hooks and workflow automation

The video mentions Cloud Code hooks that run scripts on events, such as:

  • when closing the project, run init.sh to avoid leaving it broken

A typical workflow described:

  1. Agent reads protocols/docs
  2. runs init.sh
  3. checks the task list
  4. updates task status (in progress)
  5. saves context externally
  6. marks tasks done and writes history once validation succeeds (tests pass)

Closing advice

A harness works best when the codebase itself is consistent, well-structured, and repeatable—so AI can predict and implement new features more reliably.

The video also recommends a separate methodology video: Spect-Driven Development (SDD).

Main speakers/sources mentioned (end)

Speaker

  • “I” (video author/host; name not provided in subtitles)

Sources/articles referenced

  • Oversell (tool-reduction / agent performance)
  • Anthropic
    • articles on effective harnesses
    • “How We Built Our Multi-Agent Research System”

Tools/platforms mentioned

  • Cloud Code
  • Copilot
  • ChatGPT
  • Unix commands (exact ones transcribed unclearly; described as file listing/reading/search utilities)
  • MCP (mentioned for connecting external task sources)

Original video