Build Systems, Not Code - Angie Jones, Agentic AI Foundation

Technological concepts & key ideas

• Shift from “agent prompting” to engineering “agentic systems”: The video argues that once you build agents beyond single-use prompts, you’re essentially doing software architecture—designing reusable systems with clear components and behaviors.

• Designing an agent as part of a larger system: An agent is not the whole system; it operates alongside files/tools, humans, and other agents. Example: a house hunting agent (“Relocation Scout”) ranks listings using neighborhood signals and user criteria.

• Systems thinking: Treat the agent like any other component:

  • boundaries & responsibilities
  • dependencies
  • failure modes

Agent design flow & workflow modeling

• Workflow design replaces “just a goal”:
  • Define what must happen to complete the task (not just “review this listing”).
  • Each run ends in one of: stop, retry, or escalate.
  • The workflow determines required context, what the agent handles directly, and when tools/persons take over.
• Decomposition to avoid “giant prompt” code smells:
  • Long instructions accumulate edge cases and exceptions (“giant prompts”).
  • The solution is to decompose the work into distinct jobs and improve task traceability, reasoning, and ease of change.
• Separation of concerns:
  • Decide where responsibilities belong (prompt vs skill vs script vs sub-agent).
  • Example breakdown in the Relocation Scout:
    • Normalize listings via a reusable skill / structured capability
    • Commute calculation via deterministic code/script
    • Neighborhood research via a sub-agent

Modularity & reuse

• Modularity applies to agentic systems too:
  • Agent skills as reusable components (e.g., normalization skill reused across multiple city markets).
  • Sub-agents as function-like modules:
    • do one scoped task well
    • don’t carry full session context (so they drop into other workflows)

Algorithmic thinking: when to use code vs agents vs humans

• Don’t let the model do everything:
  • Use code for deterministic tasks (exact answers like commute-time calculation, deduplication).
  • Use agents for judgment/interpretation (fuzzy reasoning, ambiguity, messy inputs).
  • Use humans for authority (final approval like booking a tour).
• Rationale: agents can be inconsistent day-to-day; code is cheaper and more reliable for exact computations.

Contracts, structured outputs, and agent memory

• Structured “contracts” between steps/systems:
  • Avoid free-form text when downstream automation needs to act.
  • Example: the ranking/decision is stored into memory with known fields like score and commute time, not trapped in a chat transcript.
• Queryable memory layer:
  • The agent writes decisions into structured memory (the speaker mentions Compendium Wiki as a memory layer).
  • Later queries can filter results, e.g., houses rated ≥4 with commute ≤15 minutes.
• Why it matters:
  • Downstream components (like the shortlist step) can read the same structured fields without human intervention.

Reliability: state & idempotency

• Design for retries and partial failures:
  • Real systems face duplicated webhooks and interrupted runs.
  • Agents must track state (what actions already happened; what’s safe to retry).
• Idempotency:
  • Re-running a task shouldn’t cause side effects to repeat.
  • Example scenario:
    • Agent emails realtor, then crashes before calendar blocking.
    • A later retry should only do the missing calendar step, not email again.
  • A key challenge: the model’s output might vary on retries, so systems must enforce safe behavior using recorded state.

Security/threat modeling

• Treat external content as untrusted input:
  • Evidence can include listing copy, forum posts, and anonymous neighborhood reviews.
  • The system should distinguish evidence vs instructions.
• Least privilege & action boundaries:
  • Constrain what the agent can do (e.g., can rank and shortlist listings freely, but cannot autonomously email/book/submit offers).
  • High-risk actions require human approval, reducing blast radius.

Maintainability as a first-class requirement

• Don’t rely on one “coding agent” to assemble other agents:
  • The result may work but be hard to maintain (e.g., messy prompts, imperfect decomposition).
• Bake maintainability into the system:
  • Include an “agent’s inD” (documentation) per system level explaining:
    • workflow
    • where policy lives
    • how to keep memory up-to-date
    • supporting resources (skills, scripts, sub-agents)
  • Target outcome: humans or agents can “cold start” in a fresh context and understand how to operate/modify it quickly.
• Testing maintainability via harness updates:
  • Example: applying an update should succeed via harnesses; failure signals the system needs better maintainability design.

Main speaker / sources

• Main speaker: Angie Jones (implied by the video title: “Angie Jones, Agentic AI Foundation”)
• Mentioned third-party/tool: “Compendium Wiki” (as an example memory layer)