AI Voice Agents Workshop - April 17, 2025

Workshop goal: build + monitor + test voice agents

The workshop focused on practical development and QA of voice AI agents, specifically:

• Building voice agents:
  • speech-to-text (STT)
  • LLM reasoning
  • text-to-speech (TTS)
  • real-time voice-to-voice
• Monitoring agent behavior in production using:
  • conversation audio
  • derived signals
• Testing/evaluating changes to avoid regressions, such as:
  • prompt updates
  • tool-calling changes
  • STT/TTS model swaps

Why voice agents are accelerating now

Key drivers discussed:

• Movement from text chatbots to more natural voice interactions
• Reduced model latency and lower costs, making latency-sensitive voice use cases more feasible

Architectural tradeoffs

• Traditional pipelines add latency at each step
• New real-time voice-to-voice models reduce handoff latency, but are:
  • harder to debug
  • less flexible

Common voice-agent architectures

Three architectures were compared:

1. Traditional pipeline (mature)

   • Speech-to-text → LLM reasoning → text-to-speech
   • Main drawback: extra latency across steps

2. Real-time voice-to-voice (lower latency, newer)

   • Single real-time model/API stream that transcribes while responding
   • Pros:
     • reduced latency
     • can convey more speech emotion
   • Cons:
     • “black box” behavior makes debugging harder
     • less flexibility

3. Full-stack platforms (“batteries included”)

   • Faster to build, but can limit customization
   • Tradeoff similar to no-code builders

Recommended engineering approach (learnings)

• Start simple: choose a narrow use case with clear ROI and decide what latency is acceptable
• Re-evaluate architecture periodically: every quarter/6 months as real-time models improve
• Bake in evaluation + monitoring from the beginning so agents don’t “go off the rails” as they scale

---

Mastra (agent framework) — voice capabilities demoed

Mastra positioning

• Mastra is an open-source AI agent framework for TypeScript
• Provides “Lego blocks” for agents, including:
  • tools
  • memory
  • tracing
  • voice capabilities
• Includes features like:
  • built-in eval framework
  • RAG
  • a local development playground
• Emphasis: Mastra is an agent framework, not a specialized standalone voice framework

Voice integration via unified API

A major demo theme was giving agents voice using Mastra’s unified voice interface. Providers implement methods for:

• Speech-to-text (STT)
• Text-to-speech (TTS)
• Speech-to-speech (real-time)

Demo 1: Speech-to-text (STT)

• Transcribing an audio file using a voice provider
• Streaming microphone audio into the agent for transcription
• Logging and confirming the resulting STT output

Demo 2: Text-to-speech (TTS) + tool use

• Demonstrated the speak method to generate audio from text
• Showed an agent capability that:
  • looked up latest sports news (Lakers)
  • generated headlines
  • sent the output to TTS for spoken delivery
• Demonstrated adjusting voice parameters (e.g., speed)

Demo 3: Text-to-speech where multiple agents “debate”

• Simulated two agents:
  • an “optimistic” agent
  • a “skeptic” agent
• Used a “process turn” style flow to iterate based on prior agent output

Added developer utility: node-audio package

Introduced a node-audio package to simplify workshop/demo tasks:

• playing audio
• recording microphone
• saving audio to files

Demo 4: Real-time voice-to-voice using WebSockets (event-driven)

A more complex implementation was shown for real-time voice-to-voice:

• Uses event-driven architecture with websockets to reduce latency
• Key methods/events described included:
  • connect / close for websocket lifecycle
  • add tools (or passing tools at agent creation)
  • updating system instructions / session config
  • emitting audio via “voice send” streaming methods
  • listening for transcription via listen events (with roles like assistant/user)
  • answer to trigger response creation
  • events for tool invocation:
    • tool started
    • tool result
  • error for handling failures

Example scenario demonstrated:

• record user audio
• upload recording (Cloudinary)
• send URL to an “agent/work” workflow
• receive a response and log metadata/status (e.g., pending)

---

ROR — monitor + evaluate voice agents (production and testing)

ROR positioning (“data dog for Voice AI”)

ROR was presented as a platform to detect whether voice agents go “off the rails” using:

• audio recordings (primary input)
• automated transcription and additional analyses
• evaluation tooling for:
  • production calls
  • test simulations before deployment

What ROR needs to evaluate calls

Integration emphasized minimal required fields:

• audio recording (ROR handles transcription and other signals)

It also supports sending extra metadata/fields, such as:

• tool invocations
• participant info (who spoke first; agent vs customer)
• custom properties for filtering and reporting (e.g., per agent, per customer)

Call dashboard experience (what users see)

ROR provides:

• transcript
• sentiment analysis
• emotional/vocal cue info (e.g., raised voice, interrupted)
• tool invocation highlights
• evaluation results (scored and explained)

Evaluation example: dental clinic appointment agent

A production-like failure was shown:

• Customer asks to book a root canal
• Agent incorrectly says they don’t offer root canals and offers cavity filling instead
• Cause: the developer forgot to define/encode treatment types in the prompt or knowledge base, leading to hallucination

Evaluation behavior:

• runs evaluators on each call (example shown: answer relevance)
• highlights failures in:
  • the transcript (marking irrelevant responses)
  • the waveform (jump directly to problematic segments)
• provides:
  • evaluation score
  • reasoning
  • breakdown of transcript segments supporting the evaluation

Reporting dashboards for scale

ROR reporting features included:

• custom reports attached to dashboards
• metrics like:
  • number of successful calls
  • average emotional sentiment
  • top reasons for call failures
  • “call topic flow” (common paths through the conversation)
• interactive drill-down:
  • breakdown by evaluator name
  • breakdown by emotion types
  • grouping by total events vs unique calls
  • click-through to relevant calls

Simulated replays / test set runs (CI/CD idea)

ROR demonstrated simulated customer runs:

• choose:
  • evaluator to test
  • agent to test
  • an existing failing scenario
• ROR simulates the call end-to-end (acting like the customer)
• many simulations can run in parallel (hundreds/thousands)

Test results include:

• pass/fail
• review of the simulated conversation

Example outcome:

• after a prompt/knowledge base fix, the simulated root canal call no longer confuses the agent; the agent requests required information instead

---

Wrap-up: access + getting started

• Repo and slides/code promised to be sent via email after the workshop
• Promotion:
  • 3 months of unlimited usage on ROR startup plan (via work.ai/master-workshop sign-up)
• Mastra quick start mentioned:
  • npm create Maestra

---

Main speakers / sources

• Shane (co-founder of Mastra; Chief Product Officer)
• Eugen (voice + cloud expert at Mastra; previously at Netlify)
• John (Netlify background; voice/cloud expertise on Mastra team)
• James (CEO of ROR; previously at AngelList; runs ROR “data dog for Voice AI” concept)
• Daniel (co-founder of ROR; CTO; previously YC startup)
• OpenAI real-time voice models (referenced in architecture discussion and demo comparisons)