Video summary

How to Choose the Right Voice for Your Voice Agent (not just benchmarks)

Main summary

Key takeaways

Technology

Main idea: choose voice like an engineering decision (not just by “it sounds good”)

Developers often pick a TTS voice by browsing a library and clicking play until it sounds right. But the video argues voice is a core part of voice agent design because it communicates:

  • Trust
  • Pace
  • Clarity
  • Brand alignment

So, voice selection should be treated as an engineering decision tied to requirements, not preference.

Layer 1: Start with the use case (not libraries, code, or public benchmarks)

The process begins with the use case: what the voice agent is doing and how users experience it.

Key point: the Cartesia voice library is organized by vertical/use case (e.g., support, gaming, health care, sales) because different jobs require different voice characteristics.

A key point from the video: benchmarks aren’t enough because they’re often lab conditions. You should benchmark against your real scenario.

Use-case guide: Stable vs. Emotive voice categories

From Cartesia docs:

Stable voices

  • Male/female variants
  • Recommended for production-grade enterprise voice agents
  • Designed to hold up under:
    • Streaming
    • Longer multi-turn conversations
    • Varied inputs

Emotive voices

  • Wider emotional range
  • Also usable in enterprise scenarios, but commonly better when expressiveness matters:
    • Companion apps
    • Gaming
    • Character/role-playing

How Cartesia curates “featured” voices (playground)

From the Cartesia playground voice library:

  • “Featured voices” are curated by Cartesia using:
    • Usage data
    • Performance metrics
    • Internal evaluation
    • Customer feedback

Users can also:

  • Browse all voices
  • View My Voices (for user-provided voices)
  • Save/bookmark/tag voices
  • Use voice previews for qualitative comparison

Voice tuning after selection: the “three knobs”

Once a voice fits the use case, you can adjust generation parameters:

  1. Speed
  2. Volume
  3. Emotion control

This is exposed via:

  • API generation config (three properties)
  • Or tags in the text input (e.g., SSML-style / transcript tags)

Important behavior note: emotion tags may be guidance, not override (Sonic 3 family)

The speaker claims Sonic 3 treats emotion as guidance, influenced by the transcript context.

  • If the text content and the emotion tag conflict, the model may ignore the tag and follow the meaning of the words.

Implication: the LLM prompt becomes critical—it should generate a transcript with sufficient emotional context, because that transcript guides Sonic’s emotion.

Prompting tips for better reliability (docs)

The video points to a prompting tips section in the docs, emphasizing techniques such as:

  • Normalization / pre-normalization
  • Controlling pacing
  • Improving spelling
  • Using tags or methods to make speech more reliable
  • Handling presentation of numbers/addresses/currency/abbreviations correctly

When to use voice cloning (instant vs. pro)

If prebuilt voices aren’t enough (the video suggests ~500 featured voices may cover most needs), consider cloning.

1) Instant Voice Clone

  • Very fast
  • Needs ~10 seconds of audio
  • Good for:
    • Prototyping
    • Testing whether cloning is worth investment
    • Iterating quickly (even for some production use cases)

2) Pro Voice Clone

  • Requires much more data (up to ~3 hours of audio/data)
  • Captures:
    • Accents
    • Speaking style precisely
  • Better for:
    • Brand continuity with an existing spokesperson
    • Rare accent/pronunciation preservation
    • Specific tone requirements
    • Character voices in games/media

Docs/API naming differences mentioned:

  • Instant clone may appear as “cloning” under voices
  • Pro clone may be managed via “datasets” (with create/update/delete management)

Production polish tools (pronunciation + pacing + spelling control)

After voice selection/cloning, the video highlights tools to close the gap from demo to production.

Custom pronunciations

Use pronunciation dictionaries to get correct reading of:

  • Proper nouns
  • Drug names
  • Brand names
  • Domain jargon

Two formats:

  • IPA (International Phonetic Alphabet)
  • Phonetic spelling (how it sounds)

Can be set via API dictionary ID or in the playground.

SSML tags / spell tags / character-by-character control

Useful for:

  • Confirmation codes
  • Serial numbers
  • Reading things character-by-character

Mentions using comma-delimited strings for slower delivery.

Evaluation strategy beyond TTS quality: test on “golden” real data + write your own evals

The video recommends testing on real-world “golden datasets” that include hard cases that break voice agents, such as:

  • Domain normalization
  • Contextual correctness

It references a blog resource about TTS evaluation failure modes (e.g., “read” vs “red”, “laid” vs “lead”).

In production:

  • Build internal evals to compare providers/services against your specific use case
  • Measure performance where it matters for your audience and failure modes

Recap: recommended staged workflow

  1. Clarify use case + target audience + expected voice behavior/failure modes
  2. Choose voice(s) that match the use case
  3. Benchmark shortlisted voices against real scenario requirements (not only public benchmarks)
  4. Optionally improve with cloning (instant or pro) and re-evaluate internally
  5. Apply production polish: pronunciations, normalization, localization, pacing/spelling controls

Main speakers/sources

  • Speaker/host: the presenter of the video (referencing “I made this video” and “we” as Cartesia)
  • Primary product/source referenced: Cartesia (cartesia.ai/voices, Cartesia Docs, and play.cartesia.ai voice library/playground)

Original video