Context Engineering for Agents

Summary of technological concepts (context engineering for agents)

Context engineering is described as the “art and science” of deciding what information goes into an LLM’s limited context window at each step of an agent’s task. It’s compared to an operating system managing RAM, where:

• LLM = CPU
• context window = RAM

For agents, context management is especially challenging because they often:

1. Run long-running tasks
2. Accumulate tool-call feedback across many iterations, increasing total token usage.

A key problem is context failures, including:

• Poisoning
• Distraction
• Curation
• Conflict

Because adding more context can increase the risk of confusion from contradictions and hallucinations, context management is critical.

---

Four strategy categories for context engineering

1) Context recording (save outside the context window)

Goal: Store information externally so the agent can retrieve it later.

Common forms:

• Draft / scratch pad notes during a single session (stored in agent state or even a file)
• Long-term memory across sessions (memory synthesized from past feedback/interactions)

Examples mentioned:

• Anthropic multi-agent explorer: researchers keep a “plan” saved for later retrieval.
• Generative agents / ChatGPT memory / Cursor / Windsurf: systems update memories based on user interaction.

Nuance: Drafts are usually session-scoped, while memory is designed to persist across sessions.

---

2) Context selection (load only what’s relevant into the window)

Goal: Bring only the relevant subset of context into the prompt.

Approaches:

• Few-shot examples / hints
• Facts and episodic/semantic memory
• Procedural memory (instructions/rules)

Practical examples:

• Rule files / .md guidelines / code agent project files pulled into context (e.g., tools that load codebase files).
• Knowledge selection via retrieval, such as:
  • Embedding similarity search and graph databases for retrieving relevant facts from large stores.
• Tool selection:
  • Agents may struggle with very large toolkits.
  • Performance can degrade with ~30+ tools and can fail at ~100 tools.
  • Suggested mitigation: RAG over tool descriptions using embeddings to retrieve only relevant tools.

Note: code-agent RAG is hard

Even though indexing + embedding retrieval can help, additional strategies may be needed, such as:

• Semantic code chunking
• File search
• Knowledge graphs
• LLM-based ranking beyond embeddings alone

---

3) Context compression (reduce token bloat)

Goal: Keep only the most useful tokens.

Techniques:

• Generalization / summarization
  • Example: automatic compression when reaching a high fraction of the context window (e.g., 95%).
  • Multi-agent papers may compress completed parts or provide subagents with compressed views.
• Pruning
  • Heuristic pruning, e.g., keep only the most recent messages.
  • Learning-based pruning, e.g., LLM-based context reduction.

---

4) Context isolation (separate context so it doesn’t pollute the main prompt)

Goal: Compartmentalize context across modules/subagents or environments.

Examples:

• Multi-agent systems
  • OpenAI’s Swarm style (“separation of duties”), with separate context windows per agent.
  • Anthropic multi-agent findings: splitting work across subagents can increase total tokens processed, improving report richness.
• Sandboxes / execution environments
  • Hugging Face Open Deep Research example:
    • A code agent generates executable code.
    • Execution occurs in a sandbox that maintains state across iterations.
    • Large artifacts (e.g., images/audio) stay outside the LLM context; only needed outputs are passed back.
• Runtime state objects
  • Use structured state (e.g., Pydantic models) to store context and retrieve only what’s needed at each stage.

---

How LangGraph is said to support these strategies

Before discussing context design, the speaker emphasizes:

• Token tracking via tracing/observability (LangSmith)
• Evaluation to ensure changes (like compression) don’t degrade agent performance

LangGraph mechanisms

State object as the basis for drafting/recording

• The state is available at every node, enabling draft-like behavior.
• Supports checkpointing for resilience during a session.
• Supports long-term memory for persisting data across sessions.

Context selection via state + long-term memory retrieval

• Long-term memory can store files and structured data.
• It can support similarity search (embeddings).
• Example mentioned: a “postal assistant” that updates memory based on human feedback (“human-in-the-loop,” positive feedback improvement).
• Tool selection can use embedding similarity over tool descriptions.

Context compression

• Provides utilities to summarize/condense message history “out of the box.”
• Allows custom compression logic within nodes.
• Mentions post-processing token-heavy tool call outputs inside the tool node.

Context isolation

• Multi-agent support (Supervisor-like and Swarm-like implementations).
• Integration with sandbox/execution environments (example: using E2B in a node with persisted state).
• Schema-based state organization (Pydantic-like) to isolate which fields are visible/used at each stage.

---

Main speakers / sources referenced (as stated or clearly implied)

• Lance (LangChain) — main presenter/speaker.
• Toby from Shopify — referenced for early “contextual engineering” discussion.
• Karpathy — referenced for the definition and OS/RAM analogy.
• Drew Brune — referenced for “context failures” framing (poisoning/distraction/curation/conflict).
• Anthropic — referenced for multi-agent explorer and multi-agent context work.
• Cognition — referenced for context compression/generalization ideas.
• Varun (CEO of Windsurf) — referenced for RAG/code-agent retrieval approaches.
• DeepLearning.AI — referenced for courses on memory and “environmental agents.”
• Hugging Face — referenced via the Open Deep Research example.
• OpenAI — referenced via Swarm.
• Jacob — referenced via a sandbox/state repository example.
• D. authors/systems referenced: Generative agents, ChatGPT memory, Cursor, Windsurf, LangSmith, LangGraph.