How to avoid cascading failures in a distributed system 💣💥🔥

Main technological concepts: avoiding cascading failures (and related “thundering herd” effects)

1) Cascading failure scenario + why it happens

• Uses a metaphor of “bison charging” to describe a sudden surge of requests that overwhelms servers.
• Example setup:
  • 4 servers behind a load balancer.
  • Requests are assigned by request ID ranges (e.g., 1–100).
  • If one server fails, the remaining servers receive subranges of the failed server’s work.
• Failure cascade:
  • If one server (e.g., S1) crashes, the load balancer redistributes its range to others (S2/S3/S4).
  • Key risk: redistribution assumes the remaining servers can handle the new load.
  • If a weaker server (e.g., S4 with low compute headroom) also becomes overloaded, it may crash too.
  • This can trigger a chain reaction: each failure increases load on the rest until the system fails for everyone.
• Framed as a race against time:
  • There is a small window (delta) before scaling/rebalancing stabilizes the system.

2) Workarounds vs “real solution”: rate limiting with per-server queues

• Workaround:
  • Stop requests for certain IDs globally (e.g., IDs 1–100).
  • Not ideal, but it reduces total outage by allowing some users to continue being served.
• “Real solution”:
  • Implement server-side rate limiting using request queues per server.
  • Each server has an explicit capacity (compute budget), mapped to throughput (example: 1 unit ≈ 1 request/second).
  • When a server’s queue exceeds its capacity threshold, it begins rejecting requests (returning failure to clients).
  • Rejection prevents overload/timeouts and helps avoid server crashes.
• Client behavior guidance:
  • Clients should not repeatedly “bombard” the system after failures.
  • Two error types:
    • Temporary errors: clients should retry later (e.g., timeouts or DB slowness).
    • Permanent errors: clients should treat the request as invalid and not retry blindly.
• Goal: keep the system stable until scaling can bring new capacity online.

3) Second problem: traffic spikes (e.g., Black Friday / viral events)

• If demand can be predicted:
  • Pre-scaling: scale out before the event.
• If demand is uncertain:
  • Auto-scaling (presented as cloud-supported and generally beneficial, with a light caution about cost).
• If traffic becomes viral/unknowable:
  • Fall back to rate limiting to cap maximum load.

4) Job scheduling to avoid “herd” effects from batch/cron work

• Example:
  • A naive cron job sending New Year emails to 1,000,000 users at the same instant creates a massive spike.
• Solution:
  • Break the job into smaller chunks (e.g., batches of 1,000 users per minute).
• Benefit:
  • Each chunk fits within normal capacity, reducing the chance of overload.
• Note on user experience:
  • Small delays may be acceptable; batch size/time window can be tuned to meet latency goals.

5) Popular-post follower notifications + jitter + approximated metadata

• Viral creators (e.g., “PewDiePie”-style) can cause spikes when followers receive notifications.
• Mitigations:
  • Batch processing (e.g., groups of 1,000 followers).
  • Jitter to spread load over time.
• Approximation optimization (YouTube-like systems):
  • Use approximate statistics (views/likes/comments) instead of exact real-time totals.
  • Why:
    • Updating exact metadata can trigger expensive database queries under extreme load.
  • Why it’s acceptable:
    • Users generally don’t need perfect real-time accuracy; approximations can greatly reduce system pressure.

6) Additional “best practices” to mitigate thundering herd

• Caching
  • Store repeated responses as key-value pairs to reduce database queries and increase throughput.
• Gradual deployments
  • Roll out in small increments (e.g., 10% at a time) and monitor before proceeding.
  • Exception: breaking changes may require parallel deployment.
• Coupling (controversial / case-by-case)
  • Example: cache authentication results (username/token) to avoid calling an external auth service every time.
  • Risk:
    • Security/correctness issues if credentials change (especially in financial systems).
  • Recommendation:
    • Cache only non-sensitive / low-risk data (e.g., profile picture), or use short TTL values to balance freshness vs load.

Main speakers / sources

• Speaker: Unspecified individual (the narrator/creator of the video).
• Referenced source: YouTube is referenced as an example system using approximate stats and jitter (not as a direct speaker).