AI Observability STACK

Technology-focused summary of the subtitles (AI Observability STACK)

Core thesis: the “metrics/logs/traces” monitoring model is inadequate for agentic/AI systems

• The video argues that traditional monitoring—often framed as the “three pillars” (metrics, logs, traces)—is a broken foundation once systems become highly complex.
• This is especially true for agentic workflows and AI execution, which can produce highly variable behavior.
• Legacy approaches are portrayed as creating disjoint silos that can obscure root causes during incidents.

Why legacy approaches fail

1. Metrics

   • Problem: Metrics are pre-aggregated, which destroys cardinality (unique-value context).
   • Result: you lose high-dimensional, per-entity detail needed to understand outages that affect specific users, tenants, or workflows.

2. Logs

   • Logs don’t correlate well across microservices.
   • Debugging becomes manual: searching text and matching timestamps by hand—“flying blind.”

3. Traces

   • Traces aren’t useful without arbitrary slicing/dicing (dynamic filtering).
   • Without that capability, trace tooling becomes an expensive interface rather than a practical debugging system.

Proposed “truth”: arbitrarily wide structured events (single unified row per request)

• The alternative model is an “arbitrarily wide structured event”:
  • Every request generates one unified row of data.
  • Each row contains 300+ dimensions of rich context, such as:
    • user ID
    • cart size
    • region
    • build number
    • feature flags
• This is presented as “practically the only way” to understand modern system behavior.

“Climbing the cardinality ladder”: scaling beyond the metrics wall

• The video emphasizes the cardinality explosion in modern systems, including:
  • Unique identifiers like user IDs and span IDs
  • For AI agents: millions of unique LLM tool call IDs
• It claims TSDB-style systems struggle at high cardinalities:
  • They can work for low-cardinality fields (e.g., HTTP status codes)
  • But around higher cardinalities (e.g., Kubernetes pod names), index pressure becomes too high
• Recommended storage approach:
  • To handle roughly ~10 million unique values, use column stores rather than TSDB-style metric storage.

Monitoring vs observability loop (known unknowns vs unknown unknowns)

• Monitoring loop (known unknowns) is described as slow:

  • Alert fires → dashboards look “green” due to lost context → manual SSH and guessing → tweak logs → repeat
  • The video’s example claims it can take up to 11 days to find one bug

• Observability loop (unknown unknowns) is described as faster and more exploratory:

  • Alert fires → because wide events preserve context, you can slice by dimensions
  • A heat map quickly reveals outliers
  • Differences are bubbled up → trace directly to the cause
• Key claim: you avoid deploying new code just to answer new questions.

System outliers: finding “the needle” even when it’s rare

• With wide structured events, you can isolate a specific slow request and understand it immediately using fields like:
  • tenant ID
  • region
  • relevant feature flag state
  • measured duration
• Argument: rare failures can be invisible to percentile-based dashboards:
  • If an outage affects only 0.3% of traffic, legacy P99 dashboards may miss it
  • With wide events, you can query for the failure even on small slices

“Agentic observability era”: debugging non-deterministic AI requires new approaches

• The video contrasts execution models:
  • Traditional software: deterministic execution tree for the same input
  • AI agents: non-deterministic behavior (same prompt can lead to different tool usage, logic paths, and failures)
• Therefore:
  • You cannot pre-aggregate an execution tree you haven’t seen yet
• Example execution described:
  • Agent plans (~240ms)
  • Tool search (~1.2s) fails
  • Agent retries (~1.4s) succeeds
  • Agent writes/reflects and produces a wrong answer
• Debugging requires tracking very high-cardinality, per-step details, including:
  • dynamic token costs per step
  • granular tool error rates
  • custom hallucination flags
• Without column stores and wide-event handling, the system becomes a black box.

New observability playbook (from “second edition” references)

The video claims a “second edition” provides a roadmap including:

• Add dedicated AI/LLM observability, not just uptime
• Implement observability governance and cost control (because tracking millions of unique values is expensive)
• Standardize on OpenTelemetry to reduce vendor lock-in
• Define SLOs (service level objectives) at massive scale
• The material is organized across 27 new chapters of industry knowledge

Main speakers / sources

• Speaker: not explicitly named in the subtitles (narration uses first-person “we”/“I want you…” phrasing)
• Source references:
  • “second edition” (of an observability/AI observability work mentioned in the subtitles)
  • OpenTelemetry (explicitly named standard)