Are You Building a Fragile AI System? AI System Design Series Ep.9

Summary

The video examines why an AI assistant can become unreliable even when its model, database, traffic, and deployment status appear healthy. The case study focuses on a slow third-party moderation API: its median latency remained stable, but its P99 rose from 300 ms to 4 seconds. Meanwhile, the assistant’s P99 increased from 6 to 14 seconds, and requests exceeding a 20-second deadline rose by about 8%.

Why a rare slowdown affects many requests

The assistant makes several dependency calls, including moderation checks on both input and output. Retries can multiply the number of moderation calls. If each call has a 1% chance of entering a slow tail, the chance that at least one of n calls is slow is:

1 − (1 − 0.01)ⁿ

With around 8–10 opportunities for a slow call, the aggregate risk is roughly 8–10%. This helps explain how a rare dependency slowdown can affect a much larger share of end-to-end requests.

Key design concepts and guidance

Sequential vs. parallel execution: Independent calls can be grouped in parallel, reducing typical latency in the example from about 2.8 to 2.3 seconds. However, parallelism does not eliminate tail latency: a slow required dependency can still dominate the overall response time.

Worker exhaustion and Little’s Law: The service has 100 workers per replica, while the moderation client can wait up to 30 seconds and retry. At 400 requests per minute (about 6.7 per second), 30 seconds of waiting can produce roughly 200 requests in flight. Adding replicas may only delay worker exhaustion if slow calls continue to occupy them.

Circuit breakers: Circuit breakers stop repeated calls to a dependency that is failing or too slow. After detecting a problem using measures such as slow-call rates or recent failures, the breaker rejects calls quickly so the application can use a fallback or return an informative error. It can then move to a half-open state to test a few calls before fully restoring traffic.

Bulkheads: Bulkheads isolate capacity so one failing dependency cannot consume all workers. The example assigns separate limits to moderation, search, and model work. These figures are illustrative; limits should be sized according to acceptable latency and expected concurrency. For moderation, two calls per request at 6.7 requests per second and a 0.3-second target imply about four concurrent calls. A fivefold reserve gives a pool of 20.

Hedged requests: If a request exceeds a threshold such as P95, send a duplicate and use whichever response arrives first. This can reduce tail latency while limiting extra work to a small fraction of requests. Do not hedge writes or other operations with side effects unless duplication is made safe.

Graceful degradation: Prepare fallback steps for important dependencies—for example, a secondary model provider, a smaller local model, cached answers, or a clear error. Other examples include treating a failed session lookup as a new session, or answering retrieval questions from the model’s built-in knowledge while clearly warning that current information may be missing. Each fallback has trade-offs and should communicate its limitations to the user.

Deadlines and time budgets: Divide the overall request deadline into budgets for individual stages, so a slow stage cannot consume time reserved for later ones. If a stage exceeds its allocation, use the degradation path rather than letting it delay the entire request.

Main speaker and source

• Main speaker: A single presenter explaining an AI system-design case study and reliability patterns.
• Source: Think in Models, “Are You Building a Fragile AI System? AI System Design Series Ep. 9.”