Video summary

I thought this feature would be easy to deploy... I was wrong.

Main summary

Key takeaways

Technology

Technological problem & goal (Kiru audio pipeline)

Kiru (a Rust-based next-gen video editor) uses an audio processing pipeline for tasks like:

  • silence detection
  • transcription
  • removing bad takes / unwanted segments

The accuracy-critical stage is the aligner, which:

  • assigns correct timestamps to transcript words in the audio
  • does this via waveform analysis

While alignment accuracy is high, the stage is too slow for product UX:

  • ~20 seconds of CPU processing per 1 minute of audio
  • Example: a 30-minute video ⇒ aligner alone takes ~10 minutes

Goal

Reduce alignment wall-clock time while keeping accuracy by:

  • splitting audio into chunks
  • processing chunks in parallel

Local prototype & concurrency design

The prototype was split into three processes:

  1. split audio into chunks
  2. worker processes run alignment on chunks concurrently (via a job queue)
  3. merge chunk results back together

Job queue / state store

  • Production idea: SQS
  • Local testing: Postgres as the message cue/state store

Postgres also tracked per-job state, including:

  • chunk list
  • completion status
  • alignment results

Concurrency testing

  • Ran across multiple nodes in a home lab
  • Used Neon as a remote Postgres provider

Measured behavior

  • Total processing time decreased as more workers were added
  • Time per minute of audio improved as well, suggesting the algorithm favors smaller chunks

Production deployment strategy decision: bursty workload + initialization latency

Several deployment strategies were compared:

  • Always-on capacity
    • good for steady services like APIs
  • On-demand / just-in-time capacity
    • good for bursty async workloads

Chosen approach (initial): AWS Lambda

Using:

  • SQS for chunk jobs
  • S3 for chunk audio data
  • Lambda workers fetching jobs and chunk data from Neon Postgres

Lambda was expected to scale to many chunks in parallel (up to configured capacity).

Failure mode: AWS Lambda cold starts break UX

Compared to the local/chunk model:

  • expected roughly ~20s per 2-minute chunk

On Lambda, observed behavior resembled:

  • ~40s-ish for a 1.5-minute video equivalent expectation

This was attributed to:

  • cold starts (new execution environments)
  • aligner startup overhead from loading large data into memory
    • estimated ~30 seconds from logs

Warm start benchmark

When execution environments were reused:

  • processing dropped to ~12 seconds

Why this was unacceptable

Even though warm starts were fast, cold starts still occurred for real users, making latency unacceptable.

Fix attempt 1: Lambda warm scaling / provisioned concurrency

Provisioned concurrency was tried to prevent cold starts.

  • Driven by “intent to start work”
  • Problem: provisioning took ~3 minutes
    • slower than cold starts
    • not practical for UX

Predictive scaling solution (intent signal → pre-warm compute)

Instead of scaling after arrival, the system was updated to scale before the workload arrives.

Intent signal in Kiru

When a user creates a project:

  • audio is extracted and uploaded to S3
  • upload requires a pre-signed URL from the API server
  • the pre-signed URL request includes metadata (duration/size)
    • used to estimate required workers

Early Lambda predictive approaches

  • attempted “pre-warm” by sending empty SQS messages
  • turned out unreliable: still experienced cold starts in some timing/chunk-count scenarios

Final solution: AWS ECS on Fargate with predictive pre-scaling

The system switched from Lambda to:

  • ECS (Elastic Container Service) + Fargate

Triggering scale-up

  • the API intent causes an ECS scale-up
  • tasks are running before chunk jobs arrive

Startup measurements

From scratch, ECS/Fargate was similar in magnitude to Lambda provisioned concurrency.

Breakdown:

  • ~10s Fargate resource setup
  • ~30s memory/data loading

Unexpected major cost

  • pulling large container images
  • Fargate lacks traditional Docker host image cache behavior

Container image optimization to reduce startup time

AWS-recommended optimizations were applied.

Reduce image size

  • multi-stage Docker builds
  • aggressive pruning
  • image reduced from ~1.4 GB to ~503 MB

Compress image

  • tested ZSTD vs gzip
  • ZSTD saved ~15 seconds more
  • gzip was chosen due to compatibility with the next step

SOCI (seekable OCI) image acceleration

  • enables starting before full image download
  • available on Linux only (not macOS)
  • workaround:
    • GitHub Actions builds the SOCI image using Linux runners

Result

  • image pull time reduced to just under ~20 seconds
  • total ECS task startup ~~1 minute

Hybrid strategy for worst-case UX

Since it wasn’t perfect for the slowest cases, a hybrid was used:

  • keep some always-on capacity
    • ~3 continuously running aligner instances
  • run them as Fargate Spot
    • about $30/month

This ensures users don’t experience unnecessary delay.

Neon (sponsor) value points highlighted

Neon features emphasized for this workflow:

  • Branching
    • instantly fork the production DB for safe debugging/testing migrations
  • Data anonymization
    • debug with LLMs without exposing user data
  • Point-in-time restore
    • recover the DB to earlier moments after mistakes
  • Works well for serverless + high concurrency workloads without overwhelming connections

Neon was used for managing job/chunk state and coordinating workers.

Main speakers/sources

  • Primary speaker: The video author/builder of Kiru (Rust video editor)
  • Sponsored source: Neon (Postgres provider)

Original video