Video summary

China Just Open-Sourced 6 Ways to Speed Up Your AI Inference (Tencent AngelSpec)

Main summary

Key takeaways

Technology

Summary of technological concepts / product features / analysis

  • What was open-sourced: A GitHub project (“AngelSpec”, by Tencent) that collects six approaches to speed up LLM inference using speculative decoding, including head-to-head benchmarking, documentation, and open-source code.

  • Important correction / framing:

    • AngelSpec is not an inference engine (i.e., not a drop-in replacement for model serving).
    • It is a training framework used to train the “drafter” model (a smaller helper) used in speculative decoding.
    • The benchmarks in the repo come from drafter models trained with AngelSpec.

Core technique: speculative decoding (30-second version)

  1. A cheap draft model predicts several future tokens ahead.
  2. The large verification model checks those guesses in one pass.
  3. If guessed tokens are accepted, they effectively provide “free speed” because verification cost is about like generating ~one token—so accepted draft progress increases throughput without changing output quality.

The six inference-acceleration approaches (two main philosophies)

Philosophy 1: Auto-regressive drafting (guess tokens sequentially)

The drafter predicts one token at a time, but cheaply compared to the verifier.

  • MTP (Multi-Token Prediction): Integrated with Tencent’s High3-style setup, using parameters set aside for guessing.
  • Eagle3: An academic head that reads the large model’s internal state to draft earlier—conceptually letting the drafter “peek” into the verifier’s thinking.

Philosophy 2: Block drafting (guess a whole block in one parallel forward pass)

Drafts multiple tokens at once in a single forward pass.

  • Dlash: Originating from a UC San Diego lab paper, adapted by others.
  • Dflare: Adaptation by Peking University + Tencent.
  • DSpark: A DeepSeek serving team hybrid.
  • D“fly” (Dfly): Tencent’s original contribution.

Key benchmark trend: block methods often accept more tokens per pass

Example acceptance numbers on a High3 ~21B active parameter model:

  • MTP: ~3 tokens/pass
  • Dlash: ~3.69
  • Dfly: ~4.79 (nearly 60% better than auto-regressive in that metric)

Why block drafting can be misleading (tradeoff explained)

Because tokens inside the block are drafted in parallel, later tokens do not benefit from knowledge about earlier accept/reject outcomes. This can create a “confidence collapse” pattern:

  • Early tokens may be accepted frequently,
  • but acceptance drops sharply toward the end of the block (e.g., acceptance falls from >70% at the beginning to <10% near the end, as described in the cited pattern).

How Dfly tries to fix it

  • Adds a small sequential/extra head after parallel drafting so later tokens gain some context about earlier tokens.
  • Tencent also uses additional architectural/training enhancements (covered in the ablations below).

Dfly ablation / engineering contribution insights

Tencent’s ablations show what drives improvement:

  • Dlash backbone: ~3.77 accepted tokens/pass
  • New backbone: ~4.40
  • + sequential head: ~4.56
  • + hidden state correction layer: ~4.60
  • + 700k extra code/math prompts during training: ~4.75

Interpretation emphasized by the speaker:

  • Total engineering/architecture gain: ~0.83 tokens
  • Training-data improvement: ~0.15 tokens, but with much less engineering effort
  • “Better data can matter” is described as a humbling lesson.

Main “real finding”: the verifier becomes the bottleneck

A systems-level observation:

  • Intuitively, if you accept more drafted tokens per pass, you should get more speed—so deeper draft blocks should help.
  • But under real load, throughput can plateau when concurrency increases:

Offline vs live behavior

  • Offline: larger drafts look strong (reported ~2.0–2.4× faster in some tests).
  • Live / real concurrency: throughput flattens, implying the large model verification step (“editor pen”) limits end-to-end throughput.

Reason

  • The drafter becomes too effective and increases the verifier’s effective workload.
  • Once verifier throughput saturates, increasing speculative depth/draft length no longer improves speed.

Sixth repo item: dcut (adaptive verification budget)

dcut is not another drafter method; it changes how much verification is spent.

  • Instead of always verifying the full draft:
    • If early tokens look confident/likely to survive, keep reading.
    • If the draft looks shaky, stop early and move on.

Observed effect at higher concurrency

  • On 64 concurrent users:
    • DFly without dcut: ~848 tokens/sec plateau
    • DFly with dcut: ~981 tokens/sec
  • Tradeoff:
    • Acceptance-rate drop reported as ~2.8% to ~8%
    • But throughput improves by about ~16% under load

Noted prior work / convergence

  • The speaker says dcut isn’t completely new:
    • DeepSeek previously had something similar as “confidence scheduled verification” (about 3 weeks earlier).
    • Later, VLM shipped a version crediting DeepSeek.

Respect for honesty in evaluation (where their winners lose)

On MT-Bench (a conversational benchmark), the speaker reports:

  • Plain MTP beats block drafters in throughput under load.
  • Example comparison at 64 concurrent users:
    • Dlash: ~1.89× base decoding throughput
    • MTP: ~2.08×

Takeaway: optimizing speculative metrics does not guarantee dominance across all workloads/benchmarks.

Critical footnote: “thinking mode” breaks acceptance

Drafters were trained on non-thinking output / regular completion mode.

When used with thinking mode (reasoning traces / chain-of-thought style / extended generation):

  • Acceptance drops from ~4.79 tokens to ~3.29 tokens
  • Reported ~31% drop in the core speculative metric that drives speedup

Implication: mode/configuration mismatch can largely eliminate the intended speedup.

Practical usage guidance (what to download/configure)

Serving High3 / “Qwen38B” / “10-centent model” / “Alibaba’s” variants

  • No training needed.
  • Drafters are already available on Hugging Face under “Angel Slim”.

Serving other models

  • The repo’s main value is as method strings / flags used by VLM:
    • MTP, Eagle3, Dlash, DSpark
    • adaptive verification (dcut-like behavior), but only under specific constraints.

Adaptive verification flag requirements (caution)

It is reported to work only when several conditions are met:

  • Works only with DSpark (as stated in subtitle wording)
  • Requires FP8 KV cache
  • Requires full Varon CUDA graphs
  • Refuses to start with pipeline parallelism (referred to as “Laura” in subtitles)
  • Not compatible with some output settings (subtitles mention “output log props”)

Concurrency-based recommendation

  • < ~8 concurrent users: use the deepest block drafter available (verifier not the bottleneck yet)
  • > ~48 concurrent users: drafting depth can hurt; the scheduler becomes the bottleneck, and fixed speculation depth may reduce performance

Warning: dynamic scheduling misconfiguration

  • An open VLM issue (from July 23) allegedly turned:
    • ~232 tokens/sec into ~24–157 tokens/sec
    • and increased request latency from ~6s to ~40s
  • Cause: a bad dynamic speculation schedule
  • Lesson: test carefully before production.

Hardware / strategic context

  • Benchmarks were run on 8× NVIDIA H20 GPUs.
  • The speaker compares H20 vs H100 compute headroom, arguing that efficiency matters more when competitors have more accelerator capability.

Ecosystem strategy

  • Tencent gives away the trainer framework and trains targets toward High3, effectively encouraging lock-in:
    • keeping the target as the default training target for drafters.

Licensing

  • AngelSpec tooling uses a custom AngelSpec license, not Apache 2.0 (even if the High3 model itself may be Apache 2.0).

Key takeaways emphasized

  1. AngelSpec = training framework for speculative decoding drafters, not an inference engine.
  2. Block drafting can raise accepted tokens per pass, but …
  3. Under real concurrency, the verifier becomes the bottleneck, causing throughput plateaus.
  4. dcut/adaptive verification improves real throughput by spending verification budget where it’s most useful.
  5. Systems performance can diverge from acceptance-rate intuition (e.g., MT-Bench throughput and thinking-mode compatibility).

Main speakers / sources (as mentioned)

  • Speaker: The YouTube narrator/analyst reviewing the repo and benchmarks.
  • Primary source/documentation:
    • Tencent AngelSpec GitHub repo (ablation/benchmark results)
    • Hugging Face Angel Slim model variants
    • VLM documentation/flags (referred to repeatedly)
  • Referenced research/labs and teams:
    • UC San Diego block diffusion drafting paper (subtitles: “Dlash” origin)
    • Peking University + Tencent adaptation (subtitles: “Dflare”)
    • DeepSeek serving team hybrid (subtitles: “DSpark”)
    • DeepSeek paper: “confidence scheduled verification”
    • VLM upstream adaptive verification mechanism attributed to DeepSeek

Original video