The Strange Economics of LLM Inference-as-a-Service

Business-focused summary: “Inference-as-a-Service” economics for LLM platforms

What the video is about (business problem)

• Several large AI companies with $500M+ annual revenue (excluding the “big three”) appear to be inference-focused rather than model labs.
• The core question: How can companies that mostly serve existing/open(-ish) models generate huge profit, while model developers have technical advantages?

---

Market categories / go-to-market positioning (4-part framework)

The video divides “inference service” providers into four categories:

• Hyperscalers (e.g., Amazon, Microsoft, Google, Oracle)
  • Sell cloud + GPUs primarily; LLM inference is an add-on.
  • Not the focus of the analysis.
• Neo-clouds (e.g., Nebius, CoreWeave, Crusoe)
  • Rent GPU clusters at low cost (training/inference infrastructure); LLM inference is important but not the primary business model.
• Full-stack inference platforms (e.g., Fireworks, Together AI, Base10, DeepInfra, Novita)
  • Primary business: package model access + serving stack + optimization + reliability.
• Specialized-chip inference platforms (e.g., Groq, Cerebras, SambaNova, Etched)
  • Primary business: improve inference speed/efficiency by using purpose-built hardware (and associated software).

---

Definition: “Inference as a Service” (operational scope)

• Inference = the model runs on input after training to generate output tokens (e.g., when you ask ChatGPT).
• Inference-as-a-service = the provider handles:
  • request processing
  • GPU coordination
  • batching
  • caching
  • low-latency token generation
• Customer action: call the provider’s API instead of operating a GPU cluster.

---

How they make money (economics model)

Providers earn from the margin between:

• cost to run inference (GPU time + memory + infra + operations)
• pricing paid per token (or per workload)

Margin can be narrow because they compete with:

• official model APIs (sometimes very aggressively optimized)
• multiple other providers serving the same model

Break-even / unit-economics example (DeepSeek V4 Pro)

Assumptions / figures from the video:

• Model weights ~ 865GB (FP4/FP8 mix)
• “Theoretical” fit: ~5 B200s, but practical overhead requires more
• Basic practical deployment: 8× B200 (e.g., one DGX B200)
• GPU rental estimate: $6 per B200 per hour
  • 8 B200 ≈ $48/hour GPU-only floor (excludes CPU/storage/network/labor)

Third-party pricing (Fireworks/Together AI) for DeepSeek V4 Pro:

• $1.74 per million inbound tokens
• $3.48 per million initial/output tokens (video refers to million “initial tokens”)
• Token ratio assumption for revenue estimate:
  • for ~1 outgoing token, client sends ~2 incoming tokens (reasonable for chat/agents/coding/document tasks)
• Implied revenue:
  • ~$6.96 revenue per 1 million “workload tokens issued”

Throughput estimate:

• “vLLM deployment on 8 B200s” can issue ~3.24M tokens/hour

Revenue result:

• ~$22.55 revenue/hour, which is below $48/hour cost → not break-even

Derived implication (hard KPI for utilization)

To break even under these assumptions, server must generate at least about:

• 2000 output tokens/second per batch load (as stated in the video)

Takeaway

Profitability depends heavily on:

• demand + utilization + token throughput
• reducing cost per token via engineering optimizations

---

Operational optimization playbook (how providers increase margin)

The video repeatedly ties competitiveness to a few engineering “levers”.

1) Prefill vs decoding optimization (speed + cost split)

• LLM inference is split into two stages:
  • Prefill (profiling / read/process the full input prompt; highly parallel)
  • Decoding (generate tokens sequentially; harder to parallel; more memory/bandwidth bound)
• Outgoing/decoded tokens cost more because the system must repeatedly use:
  • KV cache
  • weights
  • memory movement

2) KV caching (reduce repeated work)

• KV cache stores intermediate attention states for input prefixes.
• Cache miss: run entire prefix through all layers.
• Cache hit: reuse prior KV cache and compute only the new suffix.

Example pricing difference (DeepCache):

• Cache hits: 0.3625 cents per million input tokens
• Cache misses: 43.5 cents per million input tokens
• Claimed savings: ~120× cheaper

Important caveat:

• Reported token/s benchmarks often include many cache hits
• One cited benchmark: 73.7K inbound tokens/sec where ~56.3% were reused cache tokens (so “pure compute” rate is lower)

3) Batch processing (amortize overhead; raise GPU utilization)

• If only one user uses the model, GPUs stay underutilized.
• Combine hundreds of users into a batch so one set of weights serves many sequences.
• Use continuous batch processing:
  • remove completed requests
  • add new requests continuously

Tradeoffs (explicitly stated):

• larger batches can increase queue latency and time-to-first-token
• larger batches consume more KV cache memory

KPI mindset implied: balance

• throughput vs latency vs memory (bandwidth, TTFB, generation speed, queue latency)

4) Prefill/decoding disaggregation (specialize hardware)

• Prefill is compute-heavy; decoding is memory/bandwidth-heavy.
• Big providers separate these:
  • prefill workers on one GPU pool
  • decoding workers on another pool
  • send KV cache over high-speed network
• Benefits:
  • scale each stage independently
  • prevent long prefill jobs from harming decoding latency for other users

Related idea mentioned:

• “profile as a service” (Kimi Moon Sha): offload prefill of long context to reduce TTFB by shipping KV cache back.

5) Kernel-level optimization (cost per token through GPU efficiency)

• “Kernels” = small GPU programs implementing operations.
• Specialized inference kernels target:
  • specific tensor shapes, precision, model architecture, and hardware
• Key optimization method: kernel fusion
  • combine multiple steps to reduce data movement between GPU memory tiers
• Another method: tiling
  • split large matrix operations into blocks matching GPU memory hierarchy

Example given: Flash Attention

• Computes attention without fully building the huge attention matrix in HBM.

6) Why first-party model labs can win (but not always)

Model developers may know:

• tensor shapes
• MoE routing behavior
• quantization details
• communication bottlenecks

Third-party providers must reverse-engineer and adapt across:

• different GPU generations
• multiple deployment stacks

Result:

• third-party may need more GPUs, get fewer tokens/sec, and use more energy until they finish optimization.

---

Specialized-chip category: differentiated performance claims (tokens/sec per user)

Groq (Language Processing Unit / LPU)

• Pre-schedules operations/data transfer (avoids some dynamic GPU scheduling overhead).
• Uses fast onboard SRAM to reduce HBM waits.
• Example:
  • GPT-OSS 120B: ~475 output tokens/sec per user (not total cluster rate; group size unknown)

Cerebras (wafer-scale engine)

• Single chip design to minimize inter-GPU/server data movement.
• Example stated:
  • WSE-3: 900,000 AI cores, 44GB onboard SRAM, ~21 PB/s bandwidth
  • GPT-OSS 120B: ~1794 output tokens/sec per user
  • Claimed 3× faster than other third-party providers
• Limitation:
  • more complex architectures (e.g., DeepSeek V4) may not be supported “right away” due to kernel/optimization requirements.

---

Strategic/business implication (why inference services may grow)

• Trend: open-source labs increasingly specialize in model development, not global serving.
• Reasons labs avoid building global inference platforms:
  • insufficient compute to train + serve at scale
  • chip access limitations (including export restrictions for some regions)
  • every inference GPU could be one less training GPU

Training gap widening:

• model makers race to train larger models (e.g., 10T+ parameters), but inference requires different infrastructure.

Therefore inference providers capture value by:

• handling latency, reliability, availability, and elasticity
• turning demand spikes into stable economics through batching/caching/disaggregation/kernel optimizations

---

Notable actionable recommendations (implied)

If building or operating an inference API business, the playbook implied is:

• Maximize cache hit rates
  • design prompts/workflows to reuse prefixes
  • implement KV caching aggressively
• Use continuous batching
  • keep GPUs saturated while controlling queue latency and TTFB
• Disaggregate prefill vs decoding
  • ensure latency-sensitive decoding isn’t blocked by heavy prefill work
• Invest in custom kernels
  • fusion/tiling/attention kernels like Flash Attention
  • reduce data movement bottlenecks
• Pursue hardware specialization when model architectures fit supported kernels

---

Key metrics / KPIs mentioned

Example economics (DeepSeek V4 Pro scenario)

• GPU cost floor: ~$48/hour for 8× B200 (GPU rental only)
• Third-party pricing (DeepSeek V4 Pro):
  • ~$1.74 / million inbound tokens
  • ~$3.48 / million initial/output tokens
• Throughput estimate: ~3.24M tokens/hour (8× B200, batch)
• Implied revenue: ~$22.55/hour (from given pricing/throughput)
• Break-even requirement: ~2000 output tokens/sec per batch load (given assumptions)

Caching benchmarks / relative savings

• Benchmark: 73.7K inbound tokens/sec includes ~56.3% cache hits
• DeepCache pricing:
  • cache hits: 0.3625 cents per million input tokens
  • cache misses: 43.5 cents per million input tokens
  • ~120× cheaper
• Optimization benchmark narrative example:
  • 73.7K inbound / 14.8K outbound tokens/sec per H800 node
  • cited 545% margin claim (tied to their pricing and optimization)

Performance (tokens/sec per user, chip category)

• Groq: ~475 tokens/sec (GPT-OSS 120B)
• Cerebras: ~1794 tokens/sec (GPT-OSS 120B)

Token costs mentioned (DeepSeek V3/V4 narrative)

• Example token prices:
  • DeepSeek V3/R1: 55 cents per inbound and $2.19 per million outbound
  • DeepSeek V4: 43.5 cents inbound and 87 cents outgoing

---

Sources / presenters (as mentioned)

• Presenter: (Unnamed narrator/creator in subtitles) who thanks supporters and references Patreon/YouTube; no personal name given in the subtitles.
• Sponsor/source mentioned: Telnyx
• Supporters credited: Spam Maj, Chris Ledoux, Teegan, Robert Zavala, Poof, Enew, DX Research Group, Alex, Leo Halawani (and “many others”).