How Spotify runs agents across 20M+ lines of code, with Niklas Gustavsson

Key themes (agents + large-scale code automation at Spotify)

• Spotify uses LLM-driven agents (via Claude / “Claude Code” / an agent SDK) to automate a large portion of engineering work, especially:
  • Code transformations
  • Code maintenance across a huge repository landscape
• A major driver was that Spotify’s codebase growth outpaced engineer growth (reported as ~7x faster), creating a mounting maintenance burden.

Personal + early “breakthrough” moment for LLM coding

• The speaker describes switching from manual code editing to an AI workflow where the model writes most code and the user does less “last-mile” editing.
• They cite a personal breakthrough around “Opus 4 or 5” (Nov/Dec):
  • moving from “smart autocomplete”
  • to being able to tackle real problems with less prompt engineering
• They also recall an early internal “AGI-like” moment:
  • LLMs started offering insights on where coding automation was heading (pre-“claw” / early GPT era)
  • even though results weren’t yet fully reliable.

Current coding workflow using agents (“Claude Code”)

• Runs in a terminal-based setup with 5–10 terminal tabs, using panes for diffs.
• Uses multiple agent sessions in the background while working.
• Repository strategy:
  • Mostly works in large monorepos (backend monorepo claimed to be 20M+ lines of code)
  • Still has thousands of poly repos
  • They open temporary agent sessions when switching into poly repos
• Monorepo concerns (initially about performance, e.g., indexing) existed, but they report it “works amazingly well”, attributing success to Claude’s ability to read other code in the repository for inspiration.

Spotify’s infrastructure: “Fleet management” → “Honk” (agentic code change system)

Why they built it

Traditional approaches (e.g., “static analysis + transformations”) hit a ceiling because:

• Codebases have huge API surface area
• Even simple API migrations involve many edge cases
• Scripts became very large, with messy state/variable tracking

How they evolved

• Early LLM attempts:
  • tried one-shot changes by showing code to the model → poor results
• Improvements:
  • used a verification/judge pattern (early)
  • decomposed tasks into smaller steps
• Consolidation into “Honk”:
  • started more homegrown and evolved into an integrated system
  • they mention a “V2,” humorously suggesting it’s closer to V8 in reality

Honk architecture + verification loop

• Honk is described as:
  • the agent SDK running inside Kubernetes pods
  • the agent has access to a set of tools
• Tooling changes:
  • Previously, tool access was allow-listed
  • In V2, users can add their own tools “off those tools,” enabling broader agent capability

Verification

• Honk can run CI builds and can do so on Linux and macOS
• They emphasize macOS because iOS-related verification requires macOS builds
• It can go beyond compilation into simulator-level testing in some cases

Example of deeper testing

• Integrating simulator + Claude to automate:
  • Figma-to-UI implementations
  • porting TV apps from iOS apps

“Judge” removal

• They removed the explicit judge step because later models/agent reasoning improved sufficiently.
• They claim the judge was important early:
  • PR success rate improved roughly from 20–30% to ~80%
• As models/agents matured, the extra judge layer became unnecessary.

Why verification and test automation matter for scaling speed

• The speaker emphasizes a closed-loop agent workflow:
  • the agent runs the task → breaks it down → executes without constant human intervention
• They call verification/test automation the single most important piece for scaling speed.
• They describe a shift in engineering practice:
  • previously, teams were in the loop for every PR touching their components
  • as they began automerger PRs, stronger automated tests became necessary because teams wouldn’t always review changes manually.

Reliability vs speed (their stance)

• They argue reliability and speed shouldn’t be treated as a trade-off:
  • speed comes from automating quality practices
  • quality becomes encoded in tooling rather than depending on individual judgment
• They report:
  • keeping quality metrics “neutral” while improving speed
• However, it still requires ongoing investment in reliability/test automation as deployment frequency increases.

Deployment speed + feedback loop

• Spotify reports about 4,500 production deployments per day.
• Motivation: shorten time-to-feedback:
  • historically: weeks/months
  • now: about an hour in many cases
• Agents are part of enabling this faster idea validation loop.

ROI measurement and linking engineering output to user value

Reported productivity metrics

• ~75%+ improvement in PR frequency attributed to AI tooling
• ~73% of PRs attributed to AI authorship

Connecting work to user impact

• They work on mapping PRs/deployments to internal work items
• Connect to A/B tests and rollouts
• Attribute outcomes back to specific deliverables (e.g., a PR “contributed to” a DoD and then to user value)

ROI complexity at higher scales

• ROI estimation becomes harder as improvements get larger, because costs/tokens/hours must be measured more precisely.

Advice for leaders and engineers

For CTOs / engineering leaders

• Invest in foundational capabilities:
  • test automation
  • verification
  • standardization of tools/frameworks
• Standardization helps agents because inconsistent code/tooling increases ambiguity (e.g., more ways to interpret code patterns → more confusion).

For experienced engineers

• Shift from focusing on implementation time to focusing on the kinds of problems you can solve.
• Agents can enable tackling codebases that are otherwise too hard/slow to enter.

For product builders (broader audience)

• LLM tools allow non-engineers to prototype in natural language.

LLM prototyping “app store” concept

• Spotify built infrastructure enabling end-to-end prototype creation in:
  • mobile apps + backend
• Claimed “unlock”:
  • Anyone can describe an idea in natural language and Claude implements it.
• They created an internal prototype app store so others can try prototypes.
• Prototyping scale:
  • previously: weeks/months of engineering
  • now: 1–2 hours
• Adoption:
  • prototyping is “everyone,” including top executives
  • one co-CEO has prototypes there, with both engineering and non-engineering participation.

Main speakers / sources

• Niklas Gustavsson (primary guest; speaker about Spotify’s engineering workflow, Honk architecture, ROI, and verification practices)
• Interviewer (unnamed host who asks questions about Spotify’s agents, workflow, ROI, and advice)