Building resilient systems with Sam Newman

Summary

Sam Newman argues that reliable software depends less on any single architecture or tool than on understanding a system’s failure modes, business purpose, and the people who build and operate it.

Microservices: useful, but not a default

Newman describes microservices as an architecture of last resort. They can help teams operate autonomously, but they also introduce distributed-systems complexity. In his view, their defining feature is independent deployability: a service should be changeable and deployable without requiring every other service to be released at the same time. Service boundaries should generally reflect business domains rather than merely technical layers.

Microservices do not automatically improve an organization. They are one way to enable team autonomy and are most appropriate when that autonomy is a real goal. Some companies adopt them expecting a magical fix, only to create many services without clear boundaries or ownership. Newman uses Uber as an example of how services can proliferate under pressure to move quickly, then later need to be reorganized around clearer domains.

Three rules of distributed systems

Newman reduces many distributed-systems problems to three basic facts:

1. Information takes time to travel. Networks introduce latency, and developers cannot fully control the path or timing of communication.
2. The thing you want to communicate with may be unavailable. Services, machines, or network connections can fail.
3. Resources are finite. CPU, memory, I/O, queues, and connection pools can saturate. Resource exhaustion is a common cause of outages, often made worse by retries that create further load.

These rules explain why timeouts, retries, load balancing, and redundancy require careful design rather than being treated as simple fixes.

Resilience, observability, and failures

Resilience is not just about preventing failures. Drawing on resilience engineering, Newman discusses four dimensions:

• Robustness: Absorb expected disturbances.
• Recovery: Restore service after a problem.
• Graceful extensibility: Respond to unexpected situations.
• Sustained adaptability: Learn from experience and keep improving.

Resilience is a matter of degree, and added safeguards can also add complexity and create new failure modes. Choices such as multi-region or multi-cloud operation should be weighed against cost, latency, operational burden, and the business consequences of an outage.

Observability is essential for understanding distributed systems and should be designed in from the start. Useful logs and metrics, structured events, and traces can connect activity across services. Newman distinguishes raw technical signals from user-centered questions: did an important action—such as submitting a request or booking a ride—actually work in an acceptable time?

He also discusses SLIs and SLOs. Service-level indicators measure performance, while service-level objectives express the level of service the team is aiming to provide. These help teams reason about service quality more meaningfully than dashboards without clear goals.

Failures should be discussed openly. Blameless postmortems and shared incident reports help organizations learn, while psychological safety makes it more likely that people will raise concerns and adapt practices. Exercises and incident simulations can help teams prepare for situations they did not anticipate.

Retries, idempotency, and overload

A retry can be dangerous when a client does not know whether the first request succeeded. Newman uses payments and ticket bookings to illustrate how a retry might accidentally charge twice or create duplicate reservations.

A common solution is an idempotency key: the client sends a unique identifier with the request, allowing the server to recognize that a retry is the same operation. This requires support on both the client and server. Another approach is to compare a fingerprint of the request’s fields, though that can incorrectly treat separate but similar requests as duplicates.

The discussion also covers retry storms and the thundering herd problem. When a service or cache recovers, many clients may retry together and overload it again. Backoff, retry limits, caching strategies, and controlled recovery can reduce the risk.

Making failure decisions in business context

Whether a system should fail open and continue or fail closed and stop is a business decision as well as a technical one. Newman contrasts an online store that might accept an order despite uncertain inventory—then notify the customer or issue a refund—with a concert-ticket site that should avoid selling a ticket if availability cannot be confirmed. The customer impact and business priorities differ.

Policies can also change as a company matures. For example, a service might tolerate payment uncertainty while growth is the priority, but stop providing service when payment fails once profitability becomes more important. These decisions should be made for individual operations, not applied uniformly across an entire system.

AI, specifications, and software development

Newman is cautious about treating AI-generated code as a substitute for software engineering. Software development is collaborative work: code is one expression of a system, not the whole system. AI may speed up code production, but that does not necessarily address bottlenecks such as slow feedback, unclear requirements, or weak collaboration.

The discussion considers whether code or specifications should be the source of truth. Newman sees this as a spectrum: some teams treat code as authoritative, while others maintain specifications and use them to guide development. In all cases, teams need to know where the system’s intended behavior is recorded and how that behavior is verified.

He distinguishes between:

• Cognitive debt: The loss of a shared understanding of the system as people and AI tools produce work without maintaining a common mental model.
• Cognitive surrender: Approving or accepting AI output without meaningfully reviewing it.

Newman recommends using AI within clear, modular boundaries. Agents can work inside a component while people define its interfaces, understand how components connect, and verify the results. Deterministic code and tests can handle predictable steps in an AI workflow. He also advises considering vendor and model dependence rather than assuming one provider will always be available or suitable.

Books, guides, and resources mentioned

• Building Resilient Distributed Systems — Newman’s book on distributed-systems fundamentals, resilience, observability, timeouts, and related engineering practices.
• Building Microservices — Newman’s earlier book on microservice architecture.
• Growing Object-Oriented Software, Guided by Tests — Recommended for readers interested in test-driven development.
• Software Architecture Fundamentals — By Neil Ford and Mark Richards; recommended as an architecture resource.
• Vlad Khononov’s book on coupling and modular design — Recommended for thinking about modular architecture.
• David Parnas’s work on information hiding — Cited as a foundational idea for designing modules.
• Uptime Labs — Recommended for browser-based training related to incident response and operational resilience.
• Videos on the Monzo Engineering YouTube channel featuring engineering discussions with Dave Farley and others.

Sponsor products featured

• Entire: Git hosting designed for agent-based workflows, with repository mirroring and storage of agent sessions, prompts, and tool calls.
• Antithesis: System testing through adversarial simulation and deterministic reproduction of failures.
• Turbopuffer: A search system combining vector and full-text search, described in the episode as using object storage.

Main speakers and sources

• Sam Newman — Software architect and author of Building Microservices and Building Resilient Distributed Systems.
• Host of The Pragmatic Engineer podcast — Interviewer and episode narrator.
• The discussion also references ideas and work by David Woods, Charity Majors, Martin Fowler, Peter Naur, and others.