Video summary
The End of LLMs? (NCM)
Main summary
Key takeaways
Summary
The video examines two research approaches to the serial, word-by-word generation used by most language models. Its conclusion is not that LLMs are ending, but that researchers are exploring ways to make generation faster and reduce the amount of reasoning expressed as billable text.
Both papers are preprints, and their results do not make ordinary local chatbots faster today.
1. Next-Concept Modeling (NCM): Draft Broadly, Then Refine
NCM adapts a coarse-to-fine strategy used in some image-generation models:
- A tokenizer represents text as several levels of codes. The coarsest level gives a rough summary; later levels encode residual details not captured earlier.
- Nested dropout trains the coarse levels to be useful on their own.
- A transformer predicts all codes at a given level in parallel, then proceeds to the next level.
- A decoder combines the levels to reconstruct the text.
This means generation takes a pass per level rather than waiting for each word or token to be generated in sequence.
Reported Results and Caveats
- On the XSum summarization test, NCM used eight passes versus 128 for the strongest diffusion-model comparison. The authors reported a 31× speedup on one Nvidia H100, and NCM scored slightly higher on the summary metric.
- On hard Sudoku boards, a roughly 25-million-parameter NCM model solved 86%, compared with about 1% for a similarly sized word-by-word model. The word-by-word model used more than ten times as many passes.
- A competing diffusion model solved 81%, making NCM’s advantage over another parallel method much smaller.
- The speed comparisons are against similarly sized diffusion language models, not conventional large chatbots. The largest NCM generator tested had 725 million parameters.
- NCM performed worse than a diffusion baseline on translation, despite using fewer passes.
- The authors warn that a single token error can alter meaning, break code, or invalidate a Sudoku solution. Results also depend on the tokenizer and the fixed coarse-to-fine sequence.
The paper’s code and models are publicly available as research code, but the video stresses that this is not yet a practical speed-up for local chatbot users.
2. Hybrid Agents: Think in Latent Space, Act Through Text
Hybrid Agents builds on the idea behind Coconut: instead of turning every internal step into a word, a model can feed its hidden state—the internal vector of numbers it has computed—back into itself.
The approach keeps tool calls and final answers in ordinary text but performs intermediate reasoning in latent space:
- A special token switches the model into latent-thinking mode.
- A small adapter called Hybrid Link modifies the hidden state, which becomes the next input.
- After each silent step, the model checks whether the state is closest to the vocabulary entry for the token that ends latent mode. It then returns to text to answer or use a tool.
- Text, latent steps, and tool results remain in the model’s KV cache as working context.
The team trained the method on Qwen 3 models. Across 11 benchmarks—including competition math, medical exams, and web research—the authors report roughly 170–180 latent steps per problem, about one-third to one-half fewer tokens overall, around twice the inference speed, and costs roughly one-third those of their text-based agent.
Important Qualifications
- The comparisons are against text-based agents trained by the authors with GRPO, not frontier commercial APIs.
- The paper’s main text does not fully explain its cost calculation.
- The reported tests cover Qwen 3 with Python and search tools. The authors say current serving engines do not support switching between text and latent reasoning across turns, so there is no corresponding Ollama setting today.
- The method can produce a readable pseudo-text trace by mapping each latent state to its nearest vocabulary item. The authors show that this readout can look sensible, but that does not prove it faithfully represents the model’s actual computation.
Broader Analysis
The video also discusses Mercury, a diffusion-based language model from Inception Labs, as evidence that parallel text generation is being pursued beyond the lab. Diffusion systems can still require multiple passes over the whole text.
The video also highlights the hidden-token issue: OpenAI documentation says reasoning tokens that users cannot see still count toward output billing and the context window.
The presenter’s assessment is that these papers are promising research directions, especially for agent costs and parallel text generation, but their headline figures are narrow and author-reported. The Sudoku result is singled out as worth reproducing at larger scale.
Main Speakers and Sources
- Main speaker: The video’s narrator, presenting a technical analysis of the papers.
- Research sources discussed: The Stanford NCM preprint; the Hybrid Agents paper and its authors; prior work on Mercury and Coconut; and OpenAI’s reasoning-token documentation.
- Other cited sources: Inception Labs’ blog, the Hybrid Agents GitHub page, and reporting on claims about attempts to probe OpenAI’s o1 reasoning.
Rate this summary
Your feedback will help improve summaries.
Improve this summary
Reprocess with a stronger model when the summary feels incomplete or inaccurate.
Translate summary in another language
Ask questions to this video
Chat for follow-up questions, clarifications, and source-backed answers.