Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Summary (technological concepts + platform features + analysis)

Distributed inference at hyperscale

• Inference has evolved into a hyperscale infrastructure workload with very fast growth.
• A key characteristic is high egress traffic compared to many traditional large microservices.
• The talk also frames AI-serving evolution like earlier cloud evolution—driven by:
  • Increasing abstraction layers
  • Growing system complexity

From model-serving frameworks to an “inference control plane”

• The discussion references serving frameworks/engines such as:
  • vLLM
  • TorchServe
  • Triton
• It argues that the next phase is forming an abstraction layer that handles:
  • Routing
  • KV cache management
  • Profile disaggregation and decoding
  • Multi-model multiplexing
• Core focus: control plane + orchestration, rather than only kernels/optimizations.

Why agent demand breaks traditional capacity planning

• For microservices, capacity often scaled roughly linearly with users and request rate.
• For agent/LLM systems, capacity scales more like:
  • users × calls per user × tokens per call
• This depends on the model, optimizations, and hardware.
• Example call patterns:
  • Chatbots: ~1 model call per replica
  • Co-pilots: ~10–20 calls
  • Research agents: ~50 calls
  • Autonomous tasks: up to thousands of calls
• Implication:
  • Capacity planning must use elasticity
  • Planning must be load-sensitive and include access control
  • Avoid spreadsheet-style microservices planning.

Key differences between microservices and LLM inference

Request shape

• Microservices: short, relatively uniform requests.
• LLMs: 50 to 100,000 tokens
  • Compute differs substantially between prefill vs decode.

Batching requirements

• Microservices: batching can be handled at the load balancer (or optionally).
• LLM inference:
  • Needs continuous batch processing during execution
  • Throughput can drop by an order of magnitude without it.

Statefulness

• Microservices are mostly stateless.
• LLM inference is heavily stateful due to KV cache:
  • Creating it is expensive
  • Deleting it is even more expensive.

Failure modes

• Microservices: failures can often be handled by restarting and restoring state.
• LLM systems: failure handling is harder due to:
  • cold → hot latency
  • the risk of losing thousands of decoded tokens
  • resulting queue buildup
• Conclusion: the bottleneck is often orchestration, not only the model.

Distributed request path behaves like a distributed transaction

• High-level workflow:
  • gateway → router → cache search → scheduler → runtime (e.g., vLLM/SGLang) → response with SLOs
• The system relies on SLOs for:
  • time to first token (TTFT)
  • time between tokens
  • bandwidth expectations
• Because each hop can be retried/fail, partial failures are more difficult than normal RPC semantics.
• Reliability should be handled by the control plane, not only “at the edge,” because the control plane can see the whole workflow.

Scheduler planning complexity (multi-axis placement)

• Unlike microservice scheduling (CPU/memory/fault domains), LLM schedulers consider ~seven+ axes, including:
  • GPU type and heterogeneous device topology (e.g., H100/A100/B200)
  • HBM reserve
  • KV cache state
  • Model weight status (cold, warmed, cold start needed)
  • Tenant priority / multi-tenant SLO profiles
  • Workflow stage and already-spent budget (agent step progress)
  • Latency budget tied to agent program type
• Emphasis: agent-oriented planning
  • Place work to minimize fast + cheap execution
  • Avoid wasted compute if later steps fail.

Optimization framework: four quadrants

The talk proposes categorizing inference optimizations into four quadrants:

1. Avoid work
   • Skip computation via prefix caching, response caching, semantic caching
2. Divide work
   • Share computation via:
     • continuous batch processing
     • prefix decoding
     • fragmented pre-filling
     • speculative decoding
3. Move work
   • Re-route to cheaper/faster resources (e.g., smaller models, cheaper regions)
4. Postpone work
   • Use access control/queues based on priority and deadlines

• Recommendation: compare new optimizations to older methods and consider cross-quadrant interactions.

Economic analysis: optimize for successful tasks, not raw tokens

• Total cost is broader than GPU/model cost and includes:
  • retries
  • storage
  • failures
  • network
  • operational overhead
• The guidance is to optimize cost per successful task (user-relevant outcome), not:
  • cost per token
  • cost per request

Reliability + cascading failures (control-plane responsibilities)

• The failure story emphasizes feedback loops:
  • GPU degradation → latency rises → client retries → queue depth increases
  • healthy GPUs become overloaded → retries increase → regional failures
• Agent complication: KV cache means you can’t simply redirect/retry like stateless services.
• Careful design elements include:
  • Circuit breakers at routing
  • Access control that rejects vs queues indefinitely
  • Load shedding based on queue depth (not only CPU/memory)
  • Retry budgets to prevent runaway costs

Observability as the input to the control loop

• Positioning: industrial-scale inference is a distributed systems problem.
• Requirement: observability should feed a control loop, not just dashboards.
• Highlighted metrics:
  • Time to first token
  • Utilization ratio (to identify whether memory or compute is the bottleneck)
  • Success rate per dollar
  • End-to-end trace latency
• A key trade-off is described (analogized to CAP theorem):
  • Larger packets improve throughput/cost but can worsen tail latency
  • Speculative decoding improves latency but increases computation
  • Smaller models reduce cost/latency but can reduce quality, causing more errors and retries (increasing cost again)
• Objective: find the best operating point among these trade-offs.

“Output control plane”: merging orchestration into one logical layer

• Central thesis: routing, batching, caching, scheduling, and reliability can’t remain separate levers.
• They must merge into a single control-plane abstraction.
• Analogy: like Kubernetes, which consolidated scheduling/autoscaling into a control plane, inference needs the same kind of step.
• Models/resources include:
  • GPU
  • KV cache
  • tokens
  • latency
  • cost
• The control plane decides:
  • which model serves which request
  • how requests are grouped into packets

Operational lessons

• Infrastructure bottlenecks often appear earlier than model bottlenecks.
  • Failures frequently stem from scheduling/routing/capacity.
• Elasticity alone can hide problems.
  • Prefer planning under resource scarcity.
• Automated management cycles beat manual processes:
  • the platform must sense / detect / adapt continuously.
• Key shift:
  • Don’t optimize for tokens—optimize for successful task completion.
• Next-stage direction:
  • from “better models” and “faster inference”
  • to orchestration (GPU memory, cache, resources all require planning/control)
• Final framing:
  • Inference is no longer just a server problem; it’s an orchestration problem.

---

Main speakers / sources

• Nishant Gupta (Meta)
• Naman Ahuja (Meta)