Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Main ideas, concepts, and lessons

• “AI harnesses” exist to improve reliability, not just model quality.

  • Models used through paid APIs/providers behave like black boxes: the provider may change models/behavior, context limits are constrained, and output correctness can’t be fully controlled.
  • Because of these uncontrollable variables, the goal of a harness is to ensure an agent behaves correctly in a stable environment, regardless of the underlying model being “rented.”

• A harness (agent harness) is the “everything around the model” that grounds it in reality.

  • This is contrasted with the ML-world meaning (a glorified test suite), because in AI/agent engineering the meaning is different.
  • From “first principles,” the term is compared to:
    • Mountain climbing harnesses: anchor you to stability so you don’t drift off-rails.
    • Dog harnesses: prevent uncontrolled behavior/cost.

Key components of an agent harness (typical moving parts)

• Tool registry: available tools the agent can call (e.g., filesystem read/write, execute bash, or browser actions).
• Model: the LLM that the agent uses (sometimes configurable).
• Context management primitives:
  • The harness often manages/compresses context to keep the agent within limits.
• Guardrails:
  • Hard limits like max steps / max tool calls and max iterations / max attempts to stop runaway behavior.
• Agent loop (and sometimes loops around loops):
  • People may think the harness is just the agent loop, but the harness is the additional structure around it.

• Verify step:

  • After the agent claims success, the harness checks deterministically that it truly succeeded (e.g., run tests, lint, or check UI/browser state).

• Core message from the demo: “Don’t just prompt harder—change the harness.”

  • The speaker intentionally keeps the prompt unchanged and improves outcomes by adding harness features: guardrails, verification, and deterministic login handling.

---

Methodology / instructions demonstrated (as a step-by-step harness build)

Goal of the demo

Build a “poor man’s AI harness” for a browser-use agent whose job is:

• Go to Hacker News and upvote the first post.

It uses:

• A deliberately unreliable model (GPT-3.5 Turbo, from the perspective of the talk as an older model).
• Playwright for browser automation (the speaker-built wrapper class launches Chromium, creates a page, navigates, etc.).

Baseline (no real harness yet)

Start with:

• A model + prompt (task instructions).
• A browser session (Playwright).
• A tool layer that lets the agent interact with the browser.
• An agent loop that continues until the agent says “stop”.

Observation:

• The agent may lie about success (e.g., it clicks “upvote” in its UI flow but ends up blocked by login).

Harness increment 1: Add default guardrails

Add guardrails to prevent uncontrolled behavior and manage context:

• Max iterations (e.g., “if you do more than six steps, kill the run”).
• Max messages (if too many messages accumulate, compress/trim context).

Example: a naive context compressor

• Keep:
  • system prompt
  • user prompt
  • the most recent two messages
• Remove the rest when needed.

Harness increment 2: Add verification so it can’t lie

Add a verify step with deterministic checks, plus:

• Max attempts:
  • Example: if the agent fails after 3 tries, stop.

Implement deterministic verification logic, including authentication failure cases:

• If evidence shows the harness’s auto-login tool did not succeed (or failed login occurred), treat the attempt as failure.
• Detect redirects to a login URL and fail if login handling didn’t complete.

Result:

• The agent stops “claiming” success when it did not actually succeed.

Harness increment 3: Add deterministic login handling (harness controls secrets/actions)

Create a login handler called by the harness during the agent loop:

• Before traces are pushed / recorded, the harness checks the browser session’s current URL.
• If not on login page: do nothing.
• If on login page:
  • Inject credentials (from secure sources / environment variables).
  • Submit the login form programmatically from the harness, not from the model/agent reasoning.
  • Add a harness message/event like: “I’m the harness. I logged in. You’re good now.”

Result:

• The agent becomes reliably capable of upvoting after login is handled deterministically by the harness.

End-state outcome (success criteria)

The run succeeds:

• logs in correctly,
• upvotes the first Hacker News post,
• and the upvote is verifiably present on the site.

---

Why harnesses matter (stated rationale)

• Models are non-deterministic → harnesses add determinism around real-world actions.
• Use “less expensive” models:
  • Cheap/smaller models can go far when properly harnessed.
• Enterprise reliability and security:
  • The speaker references IBM’s OpenRAG project for enterprise RAG over sensitive/internal data, emphasizing that harness engineering provides enterprise-level security and controlled execution.

---

Predicted future direction (speaker’s outlook)

• 2025: year of agents
• 2026: year of harnesses
• Potential next leap:
  • 2027: dynamic on-the-fly generated harnesses
    • Agents could self-generate a harness before acting (similar to Plan mode, but more concrete/self-aware).
    • Example: agent generates a harness to buy a flight ticket, with guardrailed steps before execution.
    • Framed as a step toward AGI-like behavior.

---

Speakers / sources featured

• Speaker: Tejas Kumar (AI developer advocate at IBM)

Sources / tools / technologies mentioned

• OpenAI SDK
• Playwright (browser automation; explicitly distinguished from “Playwright MCP”)
• GPT-3.5 Turbo
• Claude Pro / Claude models (context of token-based rented inference)
• Anthropic
• Google
• Watson models / Watson
• IBM OpenRAG (open-source project)
• RAG (retrieval-augmented generation)
• Hacker News (demo target website)