Video summary

The Next Generation of Procedural Audio!

Main summary

Key takeaways

Technology

Summary: Next-Generation Procedural Audio

This video is an Audio Programmer Meetup talk about the next generation of procedural audio, focusing on procedural sound effects—all non-dialogue, non-music game audio such as explosions, footsteps, environmental atmospheres, and sci‑fi effects.

The speakers argue that procedural audio can provide adaptive, infinite-variation sound with low storage, but it currently has limited tooling for sound designers and inconsistent sound quality compared to high-quality recorded samples.

1) What “procedural sound effects” are

  • Synthesized/generated in real time and adapt to changing inputs driven by gameplay (e.g., speed, loudness, dynamics, state changes).
  • Historically, early games relied on procedural/electronic sounds because there was no practical storage for samples.
  • Procedural audio has advanced through research that formalized approaches (including physical modeling and other synthesis techniques).
  • Foley concepts exist in games too (recorded real-world performances mapped into gameplay systems), but procedural audio is positioned as a more scalable next step for worldbuilding.

2) Key benefits and drawbacks

Advantages

  • Low storage requirements
  • Flexibility/control
  • Infinite variations, including sounds that were never recorded
  • Procedural effects can be gameplay/state-driven for interactivity

Drawbacks

  • CPU cost: efficient computation is required
  • Tooling gap: sound designers lack workflows to deploy procedural models broadly
  • Inconsistent sound quality: some models sound great, others need significant work

3) Nemesis: “write once, deploy everywhere” tooling for procedural audio

The talk introduces Nemesis, a company spun out of research at Queen Mary University of London, aiming to make procedural sound effects usable at scale by sound designers and game developers.

The core product idea

A tooling pipeline that can:

  • Define a portable model interface (e.g., stereo output requirements, control parameters, presets)
  • Use templating to generate:
    • Skeleton code (DSP implementation in C++)
    • Tests ensuring the translated DSP matches intended behavior
    • Automated target projects for multiple audio/game platforms

Target deployment environments mentioned

  • WebAudio / Web (via WebAssembly + Web Audio worklets)
  • Wwise
  • Roadmap/work discussed for:
    • Unreal MetaSounds
    • Unreal native audio
    • Unity
    • VST

Implementation details emphasized

  • Model and parameters described via YAML, including:
    • control parameter types (floats/ints/bools/triggers)
    • presets
  • Automated boilerplate generation
  • Sanity checks, such as:
    • duration accuracy
    • clipping prevention
    • numerical stability
  • Supports binding generation for prototyping/testing (e.g., Python)

Demonstrated example model: an explosion

  • A procedural explosion model inspired by “Designing Sound” style ideas (Andy Farnell):
    • controllable perceptual components like rumble, hiss, and dust
    • parameterized controls affecting distance/intensity and perceived elements
  • Central claim: the same DSP behavior can be deployed across platforms.

4) Improving sound quality with a hybrid neural + procedural approach

To address inconsistent quality, the talk describes research on hybrid neural-procedural sound modeling that preserves human-interpretable controls (also linked to Queen Mary University of London).

Hybrid model structure

  1. Parameter prediction network

    • Inputs from real recordings represented like spectrogram-style data
    • Outputs control parameters for an existing procedural DSP model
    • Goal: align procedural output with recorded targets

  2. Style transfer / refinement network

    • Refines the procedural DSP output to sound more like the real recording
    • Conceptually like style transfer for audio spectrograms
    • Uses a vocoder-like step (spectrogram-to-waveform) to produce audio

How control is preserved

  • The system retains the procedural model’s slider/control parameters
  • Users can still modify parameters at runtime, while neural refinement improves realism

Demonstrated outcome (conceptual progression)

  • Real recording
  • vs pure procedural/DSP output (less realistic)
  • vs refined neural output (more realistic)

Key message: better audio quality without losing interactive controlability.

5) Q&A themes: adoption challenges and performance considerations

Common challenges companies face

  • Perception bias: “synthetic must be low quality” (influenced by historically poor synthesis results)
  • Coverage problem: not every sound has an off-the-shelf model; sometimes bespoke models are required
  • Adoption barrier: procedural audio enables “virtual worlds,” but studios must buy in and change workflows

Real-time performance and CPU budgeting

  • Neural DSP can run in real time, but games face a CPU budget trade-off
  • Mitigations discussed:
    • efficiency research
    • precomputation
    • pruning networks
    • explicit “knobs” to balance CPU usage vs quality
  • A middle-ground approach mentioned:
    • differentiable DSP frameworks (DDSP-like)
    • make a DSP model differentiable and optimize parameters via gradient descent
    • noted: improvements are possible, but their explosion case saw bigger gains from the neural neural approach
      • (as described in the talk’s discussion)

Standards/languages

  • Question about using specific music/game composition languages (examples mentioned include FA/Cmajor):
    • they found such languages useful for generating backends during prototyping
    • but they chose a custom/homegrown solution aligned to game plugin needs and parameter reporting

Integration with game engines

  • Emphasis: no need to change engine developers’ core code
  • Nemesis focuses on plugins that fit existing plugin APIs and middleware:
    • “installable plugins in marketplace” model for engines/middleware

6) Main speakers and sources

Speakers

  • Josh Rice (co-presenter): framing/abstract for the meetup and discussion of procedural audio opportunity
  • David Martella (lead engineer): deep dive into Nemesis tooling and the neural-procedural hybrid refinement

External affiliations / references mentioned

  • Nemesis
  • Queen Mary University of London
  • Related tools/companies referenced: Juce, Kronos, 11 Labs, Tsugi
  • Middleware/engines/tools mentioned: Unity, Unreal MetaSounds, Wwise, FMOD, WebAudio, VST
  • Neural audio framework concept: vocoder (spectrogram-to-waveform)
  • Inspiration reference: Andy Farnell (explosion model inspiration)

Original video