Video summary

Harness Engineering Practical Demo | Make AI Agents Reliable

Main summary

Key takeaways

Technology

Summary of the demo (Harness Engineering to Make AI Agents Reliable)

Goal

Demonstrate how to make an AI agent more reliable for a banking transfer use case using harness engineering—without changing the prompt or the underlying model.

Key emphasis: Reliability improves by adding deterministic controls and tool-level validation, while keeping the LLM setup constant (including use of an older model).

Stage 1: Initial agent (Works, but is not reliable)

Agent implementation

  • Built with Google ADK using ADK’s built-in web interface (no custom UI/API layer).

Tools provided to the agent

Simple Python functions:

  1. Check balance
  2. Find beneficiaries
  3. Transfer funds
  4. Check transfer status

Mock data

  • Includes one user account and a beneficiary list with multiple similarly named contacts (e.g., multiple “John” beneficiaries).

System prompt

  • Kept extremely simple and instructs the agent to use tools and not guess.

Failure observed

  • When asked to transfer to “John,” the agent selects the wrong beneficiary due to multiple matches and insufficient clarification.

Takeaway: Even with a reasonable prompt, LLM behavior remains probabilistic; banking-like tasks require code-based guarantees.

Stage 2: Harness layer via deterministic validation

Change approach

  • Keep prompt/model the same, but add deterministic validation into the transfer tool.

Policy service

The transfer money tool now calls a deterministic function like:

  • policyService.validateTransfer

Validation checks included

  • Beneficiary must match exactly what the user requested
  • If more than one beneficiary match exists:
    • return needs clarification
    • provide matched beneficiary details
  • Reject transfers where amount ≤ 0
  • Reject if the user account is inactive
  • Reject if beneficiary account is inactive/blocked
  • Reject if transfer exceeds available balance
  • Reject if transfer breaches daily transaction limit

Sensitive info handling

  • When asking for clarification, the demo sends only the last 4 digits of account numbers to reduce leakage of full customer data to the LLM.

Demo results

  • “Transfer 10,000 to John” → needs clarification (multiple Johns)
  • “Transfer 10,000 to John Smith” → successful
  • “Transfer 25,000 to Nick” (blocked beneficiary) → rejected

Key claim: Reliability increases significantly after adding this single harness layer—without changing prompt/model.

Stage 3: Transaction confirmation workflow

Problem addressed

  • Even with validated execution, the demo argues the ideal UX is human confirmation before transfers.

Tool splitting

transfer money is split into two tools:

  • initiate transfer
    • runs the deterministic checks
    • returns pending confirmation when successful
  • confirm transfer
    • executes only after user approval

Important detail

  • No new instructions are added to the prompt.
  • The agent’s next action is driven by the structured tool response (e.g., pending confirmation).

Demo results

  • With an ambiguous name, the agent asks for beneficiary clarification (deterministic).
  • After specifying the beneficiary, the agent requests confirmation before executing.
  • Attempts to bypass confirmation still fail because deterministic tool output requires confirmation.

Observability and monitoring (audit + session-based limits)

Audit logging

The demo adds an audit log capturing:

  • What tool the agent called
  • What inputs were used (beneficiary, amount)
  • Which validation outcome occurred
  • Whether confirmation was requested/received
  • Execution success or failure

Additional deterministic policy

  • Limits the user to at most two transfers per chat session.

Failure demo

  • A third transfer attempt is blocked (e.g., “transfer 25,000…” fails due to maximum attempts/session limit).

Key claim: Observability is essential in enterprise systems so you can trace why a decision/tool call happened and fix issues later.

What the video teaches as “harness engineering”

Wrap the agent with:

  • Deterministic validation (policy checks before tool execution)
  • Human-in-the-loop confirmation (tool workflow requiring explicit approval)
  • Privacy controls (e.g., avoid sending full account identifiers to the LLM)
  • Observability (audit logs and monitoring)

Notes: The video also acknowledges harness engineering is broader and mentions additional topics not covered, such as context management and memory management.

Main speaker / source

  • Speaker: The presenter of the demo (referred to as the host of “Applied with AI”).
  • Primary source/system: Google ADK (agent framework) + GPT-4 or “mini” (an older model as used in the demo).

Original video