This New AI Architecture Is 31x Faster (NCM)

Summary

The video examines Stanford’s Next-Concept Modeling (NCM), a language-model architecture that generates text from a rough, whole-response sketch toward increasingly detailed wording. Rather than predicting one token at a time like a conventional autoregressive model, NCM predicts a sequence of concept-code levels: early levels contain a compact outline, and later levels add detail. These “concepts” are learned compression codes, not human-readable ideas.

How NCM works

A compressor represents text as multiple levels of codes. The top level is a short, coarse representation; the bottom level has one code per word.

A standard transformer generates one level at a time, completing each level in a pass. The number of passes depends on the model’s fixed number of levels, not directly on the answer’s length. Unlike diffusion models, which repeatedly process the full set of text positions, NCM begins with very few codes and adds detail in stages.

Performance claims and tests

The paper reported that NCM matched a strong diffusion model on news summarization while running 31 times faster. The narrator notes that the paper’s speed comparisons were against diffusion models, not conventional autoregressive LLMs.

In tests on a DGX Spark, the narrator found that the released NCM model reproduced the paper’s 86% success rate on difficult Sudoku puzzles. Compared with a same-size autoregressive model, NCM was about 4× faster for a single translation request and nearly 2× faster for a news-summarization request.

With 64 requests running at once, the autoregressive model was faster. The narrator attributes this to batching: the GPU can reuse model weights across requests, making computation more prominent, while NCM processes more positions overall. The released code also reruns the full query on each pass, which particularly affects long inputs.

In the narrator’s Sudoku experiment, a separately trained diffusion model solved about a quarter of the difficult puzzles with random cell filling. Changing the decoding strategy to fill the most-confident cell first brought its result close to NCM; giving it more attempts raised its success rate to 95%. The narrator therefore suggests that cell-selection order explains much of NCM’s apparent Sudoku advantage, though these were small-scale experiments.

Limitations and takeaways

NCM’s intermediate outputs are not reliably readable: it can revise even given Sudoku clues during early passes, so the response may only make sense at the end. The narrator says conventional word-by-word models performed better on readability and quality in real-world text tasks in the paper’s tables.

Other limitations include:

• NCM uses a fixed number of levels, so even short answers go through all of them.
• The compressor can confuse names and numbers.
• The largest model discussed was under one billion parameters, leaving performance at larger scales unknown.
• The released code was described as intended for evaluation; the narrator reported no training code and issues with prompt caching.

Overall: NCM appears promising for single-user local inference, where fewer passes can reduce memory-related delays. It may be less suitable for heavily batched servers or simultaneous agent workloads. The video presents it as an interesting coarse-to-fine generation approach, while cautioning that its advantages depend on workload and that the Sudoku result may owe much to decoding strategy.

Reviews, guides, or tutorials

• Review and analysis: The video critiques the paper’s speed comparisons and examines NCM’s quality, throughput, Sudoku performance, and limitations.
• Hands-on tests: The narrator runs the released model on local hardware, compares it with a same-size autoregressive model, and tests a separately trained diffusion model on Sudoku.
• Tutorial: No step-by-step setup tutorial is provided.

Main speakers or sources

• Main speaker: The video’s narrator, who conducts the local-hardware and Sudoku experiments.
• Research source: Stanford’s Hazy Research NCM paper and released models, as described in the video.
• Other model references: Conventional autoregressive LLMs and diffusion models, including examples mentioned in the narration.