When to Build Your Own Agent Harness | Harrison Chase, LangChain

Summary

Harrison Chase, LangChain co-founder and CEO, explains how to decide whether to use an existing agent harness or build and customize one. He describes an agent as having three parts:

• Model: The underlying language model. Being able to switch models can reduce vendor lock-in.
• Context: Information supplied to the agent, such as memory, domain knowledge, and conversation history.
• Harness: The orchestration layer that determines what context reaches the model, runs tools, and feeds their results back into the agent.

How harnesses work and can be customized

A basic agent runs in a loop: the model receives a request, may call tools, receives the results, and continues. More capable agents often retain this basic structure while adding features around it.

Chase describes middleware, hooks, and plugins as ways to customize points in the loop. These can add sandbox or file-system access, sub-agents, memory, context summarization, or handling for large tool outputs. More explicit cognitive architectures—with specialized steps and checks—can still be useful for workflows that need greater guidance or control.

His recommendation is to start with a general harness, then add specialized components as the use case demands.

When to use a ready-made harness

Chase’s rule of thumb is that the more a task resembles what a model was trained to do, the more likely an off-the-shelf harness is to work well. Tasks that depart substantially from typical model capabilities, or require especially strong predictability and control, may need customization.

However, a custom system should preserve familiar, model-specific approaches for tasks that are already well represented in training. File editing is one example: models may perform best when given the editing tools and workflows they were trained to use. LangChain’s Deep Agents uses model profiles to select different implementations for these tasks, depending on which model is running.

Evaluation and observability

Chase argues that organizations should create their own benchmarks because those benchmarks define what counts as a good result for their work. Evaluations can reveal regressions and help teams compare models, harnesses, and reasoning approaches.

He presents Harbor, an open-source framework for running agent evaluations. A Harbor task can specify:

• An environment, often run in a sandbox and defined with a Dockerfile
• An instruction or prompt for the agent
• A verifier, such as code, unit tests, an LLM judge, or another agent

He also discusses LangSmith for evaluating and observing agent experiments. Alongside task scores or accuracy, teams can track latency and token use to understand cost and performance.

Observability is especially useful for debugging. Chase says many agent failures stem not from an incapable model, but from poor context. Inspecting the context sent to the model, the tools used, and the sequence of messages can help identify what went wrong. Traces provide an overview of an agent’s actions, while more detailed tracing can reveal what happened within individual steps.

The data flywheel and demo

The proposed improvement cycle is to run an agent, collect traces and feedback, identify recurring problems, and test changes to the harness, model, or context. Feedback can come directly from users or be estimated automatically using code, smaller models, or LLM judges. Thoughtful UX can also encourage useful feedback without relying only on explicit thumbs-up or thumbs-down ratings.

Chase demonstrates a LangSmith “engine” that analyzes traces, finds patterns, and proposes tasks or fixes—including changes to prompts, context, and code. He says the team also uses benchmarks to compare agents. For example, they observed Codex writing scripts to analyze traces and then incorporated that behavior into their own engine toolkit.

Key recommendations

1. Start with a ready-made harness for common tasks; customize it as the task becomes more specialized.
2. Use middleware and hooks for incremental changes before adopting a more elaborate cognitive architecture.
3. Keep familiar, model-specific workflows for in-distribution tasks such as file editing.
4. Build task-specific evaluations and measure accuracy, latency, and cost.
5. Use observability to inspect the context and tool activity behind agent failures.
6. Turn production traces and feedback into experiments and continuous improvements.

Reviews, guides, or tutorials

No product review is presented. The talk includes a conceptual guide to harness design and a demonstration of LangSmith’s trace-analysis engine.

Main speakers and sources

Harrison Chase of LangChain is the main speaker. An unnamed event host introduces him and asks questions. The video is from Sequoia Capital.