Video Summary - Agent AI System Design Explained in 27 Minutes

Agentic AI system design: production software, not a chat wrapper

The video explains agentic AI system design as production software, not just an LLM wrapped in a chat UI. An “agentic” system can reason over goals, use tools, retrieve context, maintain state, make multi-step decisions, and trigger real actions via APIs.

The speaker frames the goal as building a reliable, cost-aware, observable, secure, and safe system that works for real users beyond demos.

Core recap: what agentic systems are

Typical LLM apps: input → model → output
Agentic AI system: goal decomposition → step decisions → tool calls → inspect results → update state → continue until stopping condition

Single-agent vs multi-agent patterns

Single agent
- One primary agent owns the control loop
- May call multiple tools, but workflow is centralized
Multi-agent
- Multiple specialized agents (planning, retrieval, coding, reviewing, execution)
- Separation of responsibilities via routing logic and input/output contracts

Tradeoff: multi-agent can improve specialization/parallelism/long-running workflows, but increases coordination overhead, more failure modes, more states, and higher cost/latency.

Building blocks (main technical components)

1) Model layer + model routing

Avoid using one “best” model for every step.

Model routing:

Use cheaper/faster models for:
- intent classification
- routing
- extraction
- schema filling
- simple summarization
Use stronger reasoning models only when ambiguity/deeper reasoning affects outcomes

Structured outputs are emphasized:

JSON schema, pydantic, tool calling
Ensures downstream steps get predictable fields

Model layer should answer:

Which model to use for this step
What output contract it returns
What happens if output is invalid

2) Tools (external world interface) + strict tool contracts

Tools can include: DB lookup, CRM/calendar/payment APIs, code interpreter, search services, ticketing, Slack/Google actions, backend functions, etc.

In production, tools should behave like APIs with:

clear name/description
input/output schema
permission boundaries
timeout behavior
retain behavior
error format

Security principle: tool contracts must prevent arbitrary instruction injection (avoid open-ended natural language tool arguments).

Tool outputs must be machine-readable:

structured success results
structured errors

Read vs write pattern:

start with read-only tools
move to low-risk writes
use high-risk writes only with validation + human approval

Emerging standard: mentions MCP (Model Context Protocol) for exposing tools/resources/context, but the core idea (contracts/permissions/logs) applies even without MCP.

3) Memory and state (separate concepts)

State = current execution context (step number, collected info, tools called, tool results, confirmation status, pass/fail)
Memory = broader stored knowledge (conversation history, user preferences, past actions, retrieved doc context, summaries, long-form info)

Avoid a common mistake: stuffing everything into a vector DB by default.

Recommend data architecture based on access patterns:

workflow/state in fast stores (Redis/DynamoDB/Postgres, etc.)
retrieved knowledge via RAG/vector search (PGVector, Elastic/OpenSearch, managed KB)
long-term archives in cheap object storage (e.g., S3)

Short-term context vs long-term memory:

prevent context window bloat
retrieve only the smallest useful context per decision

Frame: memory design is data architecture—what to store, where, retention, retrieval, and what’s safe to send to the model.

Example: rescheduling a doctor appointment

Store structured workflow items (appointment ID, proposed time, confirmation)
Avoid sending full medical history when not needed

4) Orchestration (control/coordination layer)

Orchestration explicitly controls the flow, for example:

user message
intent classification
context retrieval
scope check
tool selection
input validation
tool execution
result inspection
confirmation (if needed)
final response

It also mentions: “lock the trace” and send evaluation data asynchronously.

Implementation options mentioned:

application code
graph-based frameworks (e.g., LangGraph)
workflow engines (e.g., Temporal)
other orchestration components (LlamaIndex workflows, LangChain, custom state machines, etc.)

Guidance:

Use deterministic pipelines/state machines when steps are mostly known
Use graph-based orchestration for branching, retries, loops, approval gates, and fallbacks
For multi-agent systems:
- orchestration must handle agent-to-agent routing
- expected inputs/outputs
- conflict resolution

Key message: don’t confuse autonomy with lack of structure—production agents need clear control flow.

5) Evaluation (trace-level, not just final answer)

Agent systems can fail subtly even when “no exception” occurs:

semantically wrong JSON
correct tool but wrong arguments
irrelevant/stale retrieval
skipping confirmation
refusing safe requests or complying with unsafe ones

Evaluation must be designed in from the start, including trace-level evaluation:

intent classification
retrieval quality
tool selection and tool arguments
policy compliance
confirmation behavior
final answer quality
task success

Maintain a realistic test set:

happy paths
ambiguous requests
out-of-scope
tool failures
malicious inputs
partial info
policy edge cases
escalation cases

Regression testing triggers when changing:

model
prompts
retrieval logic
tool schema

Asynchronous production sampling:

judge models / “LLM-as-judge” (imperfect)
combined with deterministic checks and human review for important flows

Metrics to track (examples):

intent accuracy
tool call success rate
invalid schema rate
retrieval hit rate
refusal accuracy
escalation rate
task completion rate
user flag rate
cost per successful task

6) Approval and policy control (human-in-the-loop for high impact)

Not every action needs approval, but high-impact actions do, such as:

email sending
deleting data
refunds
appointment cancellations
billing changes
CRM updates
running code
orders
financial transactions

Safer pattern:

model proposes
code validates
user approves
tool executes

Deterministic validation should check:

ownership
permissions
required fields
whether the user confirmed the exact action

Multi-agent note: execution must not blindly trust planning; business rules live in application code, not the agent.

Production principles tying it all together

Reliability

Make behavior predictable even when models fail.

Achieved via:

decomposition into smaller steps
contracts + schema validation
retries/fallbacks
monitoring

Treat model calls as unreliable dependencies:

timeouts, malformed outputs, rate limits, provider errors
degraded quality handling

Separate deterministic validation from model reasoning (code verifies extracted fields, permissions, arguments).

Cost and latency control

Agentic systems may require many model calls per request (intent, rewriting, retrieval, planning, tool selection, interpretation, final response, evaluation).

Strategies:

model routing + smaller models where possible
aggressively limit tokens (output size impacts cost/latency)
caching (retrieval results, policy lookups, tool metadata, stable context)
batching/asynchronous work for non-blocking tasks (evaluation/summaries)
stream responses to avoid blank UI during long tool calls
enforce scope early to avoid burning tokens/tool calls on out-of-scope requests
cost observability: tokens in/out, cost per step/conversation/successful task

Context and RAG design

Goal: pass right context, not everything.

Context sources:

user message
conversation state
app DB
retrieved documents
user profile
long-term memory
tool results

Retrieval quality matters:

chunking
metadata filters
hybrid search
reranking
freshness controls
source attribution

Security/trust separation:

retrieved documents should not override system instructions
tool outputs are treated as data (not instructions)
isolate untrusted user content from system/developer instructions

For long conversations:

use summarization/checkpoints instead of sending full history
store summaries separately from raw logs

Observability, security, privacy

Observability: log/trace every agent run:

model name/version, prompt version
step name, workflow ID, conversation ID
tool name + arguments (mask sensitive values)
latency, time-to-first-token, token in/out, costs
retries, fallbacks, feedback usage, eval scores
pinpoint failure stage (intent, retrieval, planning, tool selection/execution, policy validation, final response)

Security: treat any input touching the model as attacker-controlled:

prompt injection
indirect injection via documents
poison tool outputs
unsafe model outputs

Mitigations:

isolate untrusted content; don’t execute raw model output
least-privilege tool permissions + approval gates
avoid executing SQL/shell directly from model text

Privacy:

send minimum required data to the model
mask PII at the right time
configure retention for logs/traces/conversation archives
ensure providers/DB/observability/logins are within the “data boundary”

Main takeaway (system design checklist)

A production-grade agentic AI system needs:

model routing
strict tool contracts
explicit memory/state management
orchestration
trace-level evaluations
approval gates + policy control
cost/latency controls
context/RAG design
observability, security, and privacy

Product/learning mentions

Promotes a free “Agentic AI System Design Master Class” (recording ~2 hours).
Mentions Zen Academy and a Mastering Agentic Certification program:
- weekly live classes
- building production-level projects
- low-code/no-code and code-heavy tracks
Mentions partnerships with multiple AI/tooling companies (e.g., OpenAI, NVIDIA, Fireworks AI, Pinecone, Llama Index, LangChain, etc.).

Main speakers/sources

Aishwarya Srinivasan (creator/speaker; author of Zen Academy).

Agent AI System Design Explained in 27 Minutes

Key takeaways