You Don't Own Your AI. The Harness Does.

Core idea / thesis

• Most people focus on the AI model (GPT, Claude, Gemini, etc.), but the video argues the model is only one layer inside an agent.
• To be “sovereign” (own/control your AI behavior and data), you should focus on the harness—the layer that manages the agent’s loops, tools, context, permissions, memory, and related mechanics.

Agent architecture: the layered stack

The speaker breaks agentic AI into layers:

1. Model (“weight”)

   • Examples mentioned: GPT, cloud-based models (Gemini), and other options such as Quen/Deepseek/Llama-related choices.

2. Inference engine

   • Mentions tooling/libraries such as Llama, VLM, Ollama, MLX, etc.
   • If running on cloud, there are services/API gateways (example: OpenRouter) to access multiple models.

3. Harness (the key focus)

   • Manages:
     • Agent loops (run tasks until completion)
     • Tool calling (allowing use of browser, terminal, computer access, etc.)
     • Context management
     • Permissions / sandboxing
     • Memory (long-term storage)
     • Hooks / telemetry / checkpoints / token counts / cost tracking
     • Customization and transparency of these elements

4. UI layer

   • Different interface types: terminal (TUI), desktop app, web app, mobile app.

“Seven jobs” the harness does (most emphasized)

• Loops: keeps the agent running through multi-step tasks until completion.
• Tool calling: governs how the agent declares and uses tools (browser, terminal, computer control, etc.).
• Context management (highlighted as the hardest/most important problem):
  • The agent has a context window (working memory) that must contain the right information to reason well.
  • When the context fills up, the harness must compact it using summarization.
  • Summarization is lossy, so the agent becomes less aware over time (“summaries of summaries”).
  • Workarounds discussed:
    • Use an external single source of truth (documents/software outside the context window).
    • Or make the context window bigger (but that’s slower and more expensive).
• Permissions: what the agent can/can’t do (e.g., sandbox restrictions, file-system access, writing to disk).
• Memory:
  • Long-term memory like sessions, project instructions, skills; sometimes multiple memory systems (user memory vs project memory).
  • Treated like a database; harness decides how it’s collected and stored.
• Hooks / telemetry / checkpoints / token counts / costs:
  • Harness visibility and control here significantly affect transparency and customization.
• Customization + transparency:
  • A major criterion: you should be able to inspect what’s happening “behind the curtains.”

UI architecture and harness examples

• The speaker compares different agent UIs and harness approaches, especially terminal UIs (TUI):

  • Mentions Hermes (local model chat in terminal).
  • Mentions other terminal-based tools such as OpenHermes / OpenCode-type experiences and similar UIs.

• Introduces DeepSeek harness (the speaker’s current “go-to”):

  • Described as preview/pre-release, rapidly changing.
  • Emphasizes MIT license and open-source nature.
  • “Plugin” architecture: everything is designed to be extendable (model selection, additional behaviors).
  • Example customization via plugins:
    • Model search with pinning favorites
    • Organizing projects via folders with visual grouping/color

“Ownership” vs hosted platforms (review/analysis angle)

The video critiques proprietary SaaS agent platforms:

• You may be a customer, not sovereign:
  • Your AI behavior/data isn’t fully under your control.
  • Providers can change models/architecture without your permission.
  • Subscription pricing can change, making costs unstable.
• The speaker also frames an economic rationale:
  • AI providers may need to raise prices significantly to become profitable, which could later make usage “prohibitive.”

Checklist questions the speaker recommends

To choose a harness/platform, evaluate:

• License: MIT / Apache / proprietary?
• Model-agnostic capability: can you bring/use your own model, or only their proprietary models?
• Transparency: can you inspect loops, tool calls, and injected context?
• Control over compaction (summarization) and the context behavior.
• Permissions/sandbox control: what the agent can touch.
• Exit costs / portability: how hard is it to leave and how dependent you are on a provider?

What the speaker is building (product/tutorial focus)

The speaker presents an agent called:

• Augmentor agent (built on top of DeepSeek harness)

  • Described as having a different UI and running on a “sidecar” concept.
  • Key capabilities:
    • Computer control (agent can do essentially anything the user directs)
    • Browser automation (summarize pages; multi-step page control)
    • Audio mode (includes an example of speaking/interaction)
    • Slash commands & prompt library:
      • Use /news, etc., where saved prompt templates get injected into the input
      • Prompt improvement workflows (e.g., “English to improve English”, “improve my prompts”)
    • Multiple “agent skins”/visual profiles, including a Skin Studio
      • Ability to upload an image and customize UI colors

• Deployment/platform note:

  • Linux-only currently; MacOS/Windows soon.
  • Mentions the site: agumentoragent.com
  • Mentions community involvement: Discord, 3,600+ members.

“Active and sovereign user” direction

The speaker’s conclusion is that the future should be:

• Not passive AI users inside closed platforms,
• but active, sovereign users who own/control the agent layer (harness + UI + permissions/context behavior).
• The plan: swap models later while keeping the rest of the system under your control.

Main speakers / sources

• Main speaker: the video narrator/presenter discussing harness architecture and demos DeepSeek harness and Augmentor agent (no specific name given in the subtitles).
• Referenced systems/tools (sources/examples):
  • DeepSeek, OpenRouter, Ollama, Llama, MLX
  • Terminal UIs like Hermes
  • Agent harnesses/platforms such as ChatGPT/Cursor/Cloud Code-type offerings