Transforming Ethernet from Underdog to Champion for AI Inference

Technological Concepts & Core Problem

• Speeding up AI inference requires fast KV movement (KV = key/value cache used during transformer decoding).

• Inference traffic often traverses Ethernet that’s effectively lossy, while RDMA expects a lossless network for high performance.

• Two common inference architectures are contrasted:

  1. Mainstream (non-disaggregated)

     • Prefill + decode on 1–2 GPUs
     • Not disaggregated
     • Storage is external, accessed over the front-end lossy Ethernet
     • Often involves TCP storage traffic

  2. Large-scale enterprise (disaggregated)

     • Prefill and decode are disaggregated
     • More disaggregation across front-end + back-end networks
     • Key question: how to prioritize KV flows across multiple network segments

What the Talk Proposes / Key Feature Set

1) Make Ethernet carry KV using RDMA-style fast transfers (without congestion signaling)

• Goal: enable RDMA-style KV transfer performance over Ethernet.
• Constraint: RDMA congestion control is sensitive to packet loss.
• Observation: datacenter congestion mechanisms (e.g., ECN/DCQCN/PFC-style behavior) are described as missing/disabled in public clouds, causing RDMA performance degradation.

2) Use a software “shim” for RDMA congestion control and prioritization (no hardware changes)

• The shim is:

  • Software-only
  • Transparent to applications
  • Does not require network hardware changes

• How it works:

  • Synchronizes sender/receiver clocks with sub-microsecond accuracy
  • Intercepts QP (queue pair) creation events in RDMA applications
  • Sends shadow probes to measure one-way delay / path behavior
  • If one-way delay exceeds a threshold (beyond expected propagation time), it sends QCN-style congestion notifications
    • QCN = an RDMA-compatible congestion control mechanism

3) Add a “Ghost Buffer” to predict in-network congestion at the edge

• Motivates an edge case: receiver-side incast, where many senders contend for one receiver.
• Problem: congestion may occur before the receiver NIC fully saturates.
• Ghost Buffer behavior:
  • Measures bytes leaving the wire
  • Maintains a rate-estimator modeled as a queue
  • Decreases its counter at a rate slightly less than line rate to preserve bandwidth headroom
  • Triggers congestion early to reduce drops and sustain low latency

4) Prioritize KV traffic categories

• Focus is on ordering/prioritization among KV transfer types:
  • KV reads: GPU requests KV from cache/store (keeping these fast prevents GPU idling)
  • KV writes: transferring KV to storage/caches (e.g., from decode nodes)
  • KV demotions: cache evictions/eviction-like moves (deprioritized)
• Challenge: how to prioritize flows when the network does not understand priorities
• Approach: use the above congestion-control toolkit plus rate limiting

System Behavior (“Day in Life” of Requests / Demo Flow)

• An inference request is routed by a request router (mentioned as VLM in the demo).
• KV is fetched and prefill happens.
• KV is pushed/sprayed to servers where decode occurs.
• This cycle repeats continuously.

Central tension: inference wants both:

• TCP (for external storage / cluster integration), and
• RDMA (for low-latency, fast KV movement)

…but TCP congestion behavior and RDMA lossless assumptions can conflict.

Evaluation Results (Measured Outcomes)

1) Front-end only (non-disaggregated prefill/decode in the shown case)

• Tested in Oracle Cloud on AMD server rentals (multi-user workload)
• Metric: time to first token (TTFT)
  • 48K prompt tokens
  • 96 concurrent users
• Compared scenarios:
  • TCP only transferring KV over lossy Ethernet
  • RDMA without shim (drop-sensitive → worse)
  • RDMA with shim + control
• Reported outcomes:
  • ~8–12× improvement over TCP for TTFT (with strong percentile improvements)
  • Lower variance: results “hug” the best-case x-axis

2) Mixed prompt lengths / fairness behavior (short vs. long prompts)

• Expectation: short prompts finish first
• Without discrimination (TCP / RDMA without shim):
  • Results can become perversely unfair (short/long completion ordering can be wrong)
• With the ghost buffer / one-way delay control, the talk claims:
  • Long prompts complete ~2× longer than short prompts (matching the prompt-size ratio)
  • Improvement attributed to reduced queuing effects and better congestion fairness

3) Disaggregated prefill + decode (front-end to back-end)

• Prefill runs on one side, decode on the other, with KV transfers across disaggregated components
• Metric:
  • TTFT
  • End-to-end latency for multi-turn agentic workloads
• Reported improvements:
  • ~2.5–3× improvements in relevant cases
  • Larger gains at higher percentiles

Product / Architecture Conclusion

• Delivering KV quickly is essential to prevent GPU idling during inference.
• Core thesis: stop relying on “hope the network cooperates,” and instead:
  • Create headroom and eliminate packet drops on front-end Ethernet using:
    • Shim-based one-way delay congestion control
    • Ghost Buffer for edge/in-network incast prediction
    • Bandwidth control / slicing concepts (presented as ongoing work, including predictable bandwidth slicing)

Main Speakers / Sources

• Speaker (primary):

  • Kamran Ahmed is mentioned as a collaborator/demo contributor
  • (The talk speaker is not explicitly named in the subtitles.)

• Referenced / related sources:

  • 2018 paper on software-only end-to-end clock synchronization for congestion measurement/control (name not provided in subtitles)
  • RDMA congestion control standard: QCN
  • Mentions of:
    • Cisco Nexus 5000 “proxy queue/virtual buffer” style concepts
    • A 1990s “virtual buffer” historical idea