Why Inference Matters 12x More Than Training | YC Paper Club

Summary

The central argument is that inference deserves at least as much attention as training. The opening speaker cited DeepSeek figures of roughly 2.8 million H800-hours to train a model and about 43,000 H800-hours of inference per day. On that basis, annual inference compute could be about six times training compute; over a two-year model lifetime, inference could account for roughly 12 times as much computation.

The talk argues that inference efficiency affects both serving costs and product margins, and that published model comparisons should account for inference compute—not just training compute.

How inference works—and where performance is lost

A request passes through several stages: routing to a data center and GPU, tokenization, scheduling and batching, prefill of the prompt, decoding of output tokens, and conversion back to text. Tool use adds further complexity because the system must represent tool instructions correctly and handle calls reliably.

The two main compute phases have different characteristics:

• Prefill processes the prompt and is compute-intensive.
• Decode generates tokens one at a time and is often limited by memory bandwidth. The speaker contrasted much higher prefill throughput with lower decode throughput, describing decoding as a major bottleneck.

Important service metrics include time to first token (TTFT), time per output token, and throughput per GPU at a specified service level. Cache behavior has a large effect: repeated prompts and agent workflows can reuse cached key-value (KV) data, while unrelated conversations often cannot. Routing requests to a GPU that already holds the relevant KV cache can therefore improve performance as well as load balancing.

The opening talk grouped inference improvements into three broad areas:

1. Model innovations: Techniques such as mixture-of-experts, state-space models, multi-head latent attention, quantization, and distillation can reduce the work or memory needed to serve a model.
2. Kernel and runtime innovations: Examples include FlashAttention, PagedAttention, and RadixAttention, which draw on ideas from GPU programming, databases, and operating systems.
3. Serving-harness innovations: Routing, speculative decoding, recursive language models, and context compaction can avoid unnecessary computation or reduce cache requirements.

Dan Fu: The lifecycle of an inference request

Dan Fu of Together AI walked through the serving stack from API request to returned token. He emphasized that a production service must jointly manage routing, cache locality, tokenization and tool calls, batching, scheduling, GPU execution, and cache retention. Scheduling involves trade-offs: maximizing throughput across users can differ from minimizing an individual user’s response time.

He highlighted three optimization areas:

• GPU kernels: Small operations such as normalization and softmax can underuse hardware when run as separate kernels. Together is exploring agent-driven, workload-specific kernel optimization, using real traffic patterns and production measurements to guide improvements.
• Parallel execution: Large models may not fit on one GPU, so tensor, pipeline, or data parallelism is needed. Newer systems that connect many GPUs with high-bandwidth links can support larger-scale execution, but the speaker stressed reliability: a failure in a large multi-GPU replica can disrupt the whole workload.
• Quantization: Storing weights at fewer bits reduces memory use and can improve throughput, but may affect model quality. Together described a training-free quantization recipe for a MiniMax model that reportedly preserved quality close to the original FP8 version, and stressed testing tool calls, benchmarks, and end-to-end agent behavior before deployment.

Jeff Morgan: Open-model tokens can vary by provider

Jeff Morgan, co-founder and CEO of Ollama, argued that using the same open-model weights does not guarantee the same results across inference providers. Differences in quantization, hardware, inference engines, sampling, model versions, and tool-call handling can all affect quality. He cited research comparing 31 Llama endpoints and said the gap among providers serving open models can be larger than the gap between open and closed models.

This inconsistency is especially risky for agents: small errors across a chain of calls can compound and derail a task. Morgan outlined several ways to monitor providers:

• Run benchmarks and end-to-end evaluations, though these can be costly and slow.
• Test tool-call behavior using dedicated validation sets.
• Compare output distributions with reference models.
• Track token log probabilities to detect changes in a provider’s serving stack.
• Monitor aggregate request patterns and tool-call success rates without retaining user data.

He also emphasized using customer-specific, non-public test cases to catch failures that public benchmarks might miss. He expects developers to combine local and cloud models, while still needing reliable ways to test both.

Joe Fioti: Compiling models through search

Joe Fioti, co-founder and CEO of Luminal, presented a compiler approach based on equality saturation. Rather than relying on large collections of hand-maintained heuristics, the compiler applies small equivalence rules to build a search space of possible programs, then selects an implementation based on measured performance.

Luminal reduces models to a small set of core operations and uses an intermediate representation designed to work across different hardware backends. Its representation is described as PyTorch-complete but not Turing-complete, which limits the search problem and allows some performance bounds to be calculated analytically. The compiler can also measure candidate kernels on actual hardware rather than depending entirely on a predictive cost model.

Fioti said the approach uses bounded search—often genetic algorithms—and seeks a strong local solution rather than an impractical global optimum. Because compiled code may run many times after compilation, spending extra time searching can be worthwhile if it improves inference speed. The compiler focuses on model forward passes and is intended to complement serving engines such as vLLM and SGLang, rather than replace their higher-level serving features.

Liangsheng Yin: SGLang, caching, and agent workloads

The final presentation focused on how agent workloads make shared context and memory management increasingly important. SGLang’s RadixAttention stores shared prompt prefixes in a radix-tree structure so that agents or requests with common context can reuse cached computation.

The speaker described further work on scheduling and memory:

• Overlap scheduling prepares work for a later batch while the GPU executes the current one, reducing idle time. The speaker reported up to a twofold throughput improvement in some Blackwell configurations.
• Hierarchical caching can keep tokens across GPU memory, host memory, and storage, reducing the cost of reloading context. In experiments, the speaker said this reduced time to first token by up to 56%.
• Unified memory management is designed to handle different types of context and model state without reserving fixed-size pools for each. Memory compaction reorganizes fragmented physical memory while preserving virtual mappings.

The broader point was that longer contexts, repeated agent calls, and models with different attention or recurrent-state designs require cache systems that can share and manage memory flexibly.

Resources mentioned

The opening speaker recommended a lecture from the CS336 “LLMs from Scratch” course and lectures by Ryner Pope and Dwarkesh. The event also included technical Q&A on quantization quality checks, GPU interconnects, compiler search, and model evaluation.

Main speakers and sources: The opening overview was presented by a Y Combinator speaker identified in the discussion as François. The speakers were Dan Fu (Together AI), Jeff Morgan (Ollama), Joe Fioti (Luminal), and Liangsheng Yin (SGLang/RadixArk).