NCM matou o LLM: dessa vez, parece que é real

Summary

The video examines Next Concept Modeling (NCM), a text-generation approach from Stanford’s Hazy Research that challenges the usual method of generating text one token at a time. The paper reports speedups of up to 31×, but the video emphasizes that this figure comes from a specific comparison and should not be interpreted as a general advantage over ChatGPT or production LLMs.

How NCM works

• Autoregressive LLMs generate text sequentially: each token is predicted using the tokens that came before it. This creates a sequence of dependent steps, even when the GPU performs many computations in parallel.
• Diffusion text models generate multiple positions together, refining a masked sequence over repeated passes. This can reduce sequential generation, but revisiting the whole sequence is costly, and using fewer refinement steps can harm quality.
• NCM uses a transformer to predict hierarchical codes rather than the next word. It starts with a short, rough representation, then adds more positions and detail at successive levels. Codes at each level are generated in parallel.

The system has two main components:

1. A multiscale tokenizer encodes text into hierarchical codes and can reconstruct the text. It reportedly achieves over 99% accuracy on most tested tasks.
2. A transformer generates the codes level by level, after which a decoder converts them back into text.

A German translation example illustrates the refinement process: the initial output is garbled but becomes a correct sentence after 11 steps.

Speed claims and results

The reported 31× speedup comes from a summarization test on the XSum benchmark, comparing NCM with the diffusion model Aurora—not with ChatGPT or an optimized autoregressive model. In that test, NCM used 8 model evaluations compared with Aurora’s 128, on one H100 with a batch size of 16, while achieving comparable or slightly better quality. The video also mentions speedups of 21× and 10× in other tests.

Strengths, limitations, and availability

• Sudoku: NCM performed well on a task with global constraints. On difficult puzzles, the video reports that the autoregressive model solved 1%, the best diffusion model solved 81%, and NCM solved 86% using 12 passes.
• Translation: Without extra techniques, NCM scored below the autoregressive model (22.6 versus 25.2). Generating multiple candidates and selecting by consensus improved results, but increased the process from 11 to 132 steps.
• Scale: The largest generator tested had 725 million parameters, much smaller than leading models.
• Fixed hierarchy: Each task uses a defined code hierarchy, so even short outputs pass through all levels.
• Reconstruction risk: If the tokenizer drops a name or number, the generator may not be able to restore it.

The code is open source under the MIT license, with task-specific research checkpoints on Hugging Face for summarization, translation, Sudoku, and other tasks. These are not presented as a general-purpose assistant to install and chat with.

Bottom line

NCM does not “kill” LLMs—it is itself a transformer-based language model. Its significance is that it questions whether text must be generated strictly one token at a time. The video recommends evaluating it on the same hardware and metrics as alternatives, especially on difficult cases involving names, numbers, and long outputs.

Guides and demonstrations covered

• A walkthrough of the autoregressive generation bottleneck, diffusion, and NCM’s hierarchical-code approach.
• Examples of translation refinement and Sudoku performance.
• Guidance for interpreting the speed benchmark and comparing models fairly.
• Information on downloading the research code and checkpoints.

Main speakers and sources

• Main speaker: The video’s presenter from the IA em 6 minutos channel.
• Research source: The NCM paper and experiments from Stanford’s Hazy Research, including comparisons with the Aurora diffusion model.