The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Summary (Technological concepts + enterprise production playbook)

Sandipan Bhaumik (Databricks) explains an AI production playbook derived from lessons learned while building and scaling data/AI systems and helping regulated B2B customers move from demos to enterprise deployments. He argues that many AI projects fail after launch because teams optimize early for model selection and demo polish rather than building systems that are measurable, traceable, and governable.

Why demos fail after deployment

Across customer conversations, the typical workflow was:

1. Debate and choose a model (e.g., GPT vs Claude)
2. Build features in controlled demo conditions
3. Launch to production once leadership signs off
4. After weeks, users ask “what is AI doing?” because the behavior doesn’t match expectations

He attributes this to three recurring gaps:

• Observability gap: Without tracing/visibility into every AI decision, production debugging and regulatory requirements can’t be met.
• Evaluation gap: Teams discuss accuracy/latency/etc., but don’t define the exact business-critical success metrics and how to measure them continuously.
• Governance gap: Little accountability exists when AI fails (who owns data assets? who responds at 3am? what if the agent gives nonsense?).

Five-pillar framework for enterprise AI agents

The playbook proposes five pillars to plan before starting projects (and to build gradually, ideally in sequence).

1) Evaluation (define success + automated testing)

• Treat evaluation as a specification before touching code or models.
• Define measurable business targets (e.g., accuracy thresholds, allowable false positives, deflection rate for simple queries).
• Build an evaluation dataset (often from real human support answers and edge cases).
• Create an automated evaluation pipeline that:
  • sends test questions to the agent,
  • compares responses to the evaluation set (“golden” data),
  • reports performance against defined metrics,
  • supports ongoing online evaluation in production.

Evaluation layers (architecture)

• Layer 1: Deterministic checks
  • format validation,
  • regex-style checks,
  • PII detection,
  • basic intent classification.
• Layer 2: Non-deterministic semantic checks using LLM-as-judge
  • a secondary model evaluates groundedness/safety/relevance
  • and scores outputs using prompts configured for judging.
• Layer 3: Behavioral evaluation
  • tool-call correctness,
  • loop detection,
  • retry behavior,
  • and cost/performance impacts (e.g., duplicate database/API calls that are acceptable in demos but expensive at scale).

2) Tracing / Observability (trace every decision)

Collect traces for:

• intent classification
• tool/database/API calls
• RAG/policy retrieval (e.g., vector DB)
• reasoning steps and guardrail checks
• response generation

Example: if a banking chatbot makes duplicate API calls or uses outdated policy documents, tracing enables diagnosis and corrective actions.

This also supports online monitoring and fallback strategies:

• retries capped at a threshold
• escalations to human review if failures persist

3) Data foundation (question data + tracking data)

“Data foundation” splits into:

• Question data: data needed to answer user questions (pre/post training data, API-connected data, RAG content)
• Tracking data: tracing/observability data used for auditing, monitoring, and running LLM-as-judge on traces

Databricks architecture components used to support this foundation include:

• Apache Spark / MLflow / Delta Lake (under the hood)
• Delta Lake for table-like structured data on raw cloud storage
• Unity Catalog for centralized governance (permissions, metadata tagging like PII columns, descriptions for context)
• additional tooling (e.g., AI app building, text-to-SQL, and observability use cases)

He also emphasizes centralized collection of tracing data even when agents run across multiple frameworks/platforms (mentions CrewAI, LangChain):

• one shared location to serve operational dashboards, support monitoring, and LLM-as-judge-style evaluation

4) Multi-agent orchestration patterns

When moving from 1 agent to many, complexity increases exponentially. Three orchestration patterns:

• Orchestrator-worker
  • centralized orchestrator controls dispatching work to specialized agents
  • easier logging/debugging
• Choreography (event/message-bus driven)
  • agents run autonomously in parallel and react to events
  • reduces latency vs orchestrator bottlenecks
• Human-in-the-loop
  • escalate when confidence falls below a threshold
  • or when performance crosses risk thresholds

(He references a separate deep dive video on orchestration, including scaling and fault tolerance patterns.)

5) Governance (regulatory + operational controls)

Governance includes:

• audit trails / traceability of actions, users, requests
• pre-validation (e.g., PII checks; rejecting unsafe content)
• prompt versioning as change management
  • treat prompt edits like code changes
  • track rationale and failure cause
• model change management
  • upgraded models must be re-evaluated using enterprise-specific evaluation datasets (not just vendor benchmarks)
• security controls
  • preventing prompt injection or malicious behavior

Case study: retail banking chatbot (POC vs production success)

He contrasts a failed earlier POC with an improved approach.

• Client issue: ~20,000 chatbot calls/month
  • ~60% were simple queries (like account balance and overdraft questions) suitable for automation
• Earlier POC: ~85K over 6 months, did not succeed
• After applying the framework: target AI handles ~60% of user queries reliably

Timeline highlights (weeks)

• Weeks 1–2
  • build evaluation layer (collect ~200 real human answers)
  • define success metrics (e.g., 60% deflection target, ~85% accuracy target)
  • automate evaluation pipeline
• Week 2
  • build foundational layer with proper API/data connections + tracing
  • ensure issues like duplicate API calls can be detected
• Weeks 7–8
  • decide model using the evaluation dataset
  • integrate evaluation + observability + tracing into production

Production outcome examples

• When banking policy changed (interest-rate/overdraft related), customer feedback dropped because the agent used outdated policy content from the vector/RAG store.
• Tracing and monitoring detected stale embeddings/document updates; the issue was then fixed.

“Production incident playbook” (how to respond when agents fail)

A structured operational procedure:

1. Detect via eval dashboard
2. Diagnose via tracing
3. Contain
   • e.g., prompt rollback/versioning
   • route to human
   • in multi-agent contexts, use fault-tolerance patterns like saga/compensation/circuit breaker
4. Fix using an evaluation test case library
   • update the eval suite so the regression is caught next time
5. Integrate with ITSM
   • alert the right owners at the right time

Key takeaways / actionable “what can you do tomorrow?”

• Define business success and what “good answers” look like.
• Build an initial evaluation dataset and automated pipeline (even with simple Python) to compare live outputs to the dataset.
• Treat the evaluation dataset (“test case library”) as a living system that grows; add governance:
  • categorize test cases by issue type (e.g., security/login issues),
  • document prompt/model changes with rationale,
  • manage cost by running full behavioral evals only on major merges (subset during prompt changes, full suite on main).

He also mentions providing artifacts via QR code (templates for evaluation checklists, guides for setting up tracing with open-source technologies) and links to his LinkedIn/newsletter.

Main speakers / sources

• Sandipan (Sandy) Bhaumik
  • Technical Lead for Data and AI, Databricks
  • formerly Principal Architect for Data and AI, Amazon Web Services