Video summary

If you like Open Models, I'm sorry

Main summary

Key takeaways

Technology

Summary

Theo argues that open-weight AI models, which he strongly supports, are falling further behind closed models in practical capability. He recalls DeepSeek’s late-2024 releases and R1 as a moment when open-weight models seemed to match or exceed leading closed models on some tasks. In his view, that parity has since faded, especially on long, complex coding tasks that require sustained coherence, planning, and coordination of sub-agents.

His central argument rests on three factors:

  1. Unequal access to compute: Theo says frontier-model progress continues to benefit from increased GPU resources. He argues that US-based closed-model labs can expand their compute faster than mostly Chinese open-weight labs, widening the gap over time.

  2. Harder-to-replicate training data and distillation: Theo describes distillation—using outputs or interaction histories from stronger models to train other models—as an important way for open labs to catch up. He argues that frontier labs’ efforts to limit distillation, along with the difficulty of collecting useful agent-use histories, have made this approach less effective. He also suggests that some resellers of discounted model access may collect prompts and responses for training. This is presented as his interpretation, not as independently established evidence in the video.

  3. Possible recursive improvement: Theo discusses reports that AI is helping researchers work more effectively on AI development. He argues that if leading labs use models to accelerate their research while restricting access to those capabilities, open-weight models may be unable to reproduce the resulting progress. He stresses that he does not believe full autonomous self-improvement, or “takeoff,” has occurred.

Evidence and Comparisons

  • Practical coding capability: Theo says the hardest-to-measure gap is performance on long-running, messy tasks involving planning and sub-agents. He considers leading closed models substantially stronger in this area than open-weight alternatives.
  • Benchmark reliability: He discusses the DeepSwi benchmark and a modified version called Max Bench, which changes tasks slightly to test whether models can generalize beyond familiar examples. He reports that many models scored lower on the modified tasks, with several open-weight models seeing large drops. He interprets this as evidence of benchmark overfitting or contamination, while acknowledging that benchmark scores alone do not capture all capabilities.
  • Usage in his coding product: Theo says open-weight models accounted for under 1% of turns and tokens in T3 Code. He also reports that the share of installs that had used an open-weight model fell from 16% to 6%. He uses this data to argue that developer adoption for real work is much lower than public interest or free-tier traffic might suggest.
  • Limits of benchmark rankings: He criticizes comparisons based solely on aggregate scores, arguing that they miss abilities such as sustained coherence and effective delegation.
  • Possible exceptions: Theo remains impressed by some open-weight releases and hopes new training methods, more compute, or better distillation techniques will reverse the trend. He singles out DeepSeek as a source of hope.

Theo’s conclusion is pessimistic, but he explicitly frames it as a prediction he hopes will be wrong. Given the widening compute gap, restrictions on distillation, and possible AI-assisted research acceleration, he does not expect open-weight models to remain competitive for demanding real-world work without major changes.

Sponsor Segment

The video includes a Blacksmith CI sponsorship. Theo says its CLI lets his coding agents run tests in remote test environments without pushing a pull request, easing local CPU and CI bottlenecks. He reports that this increased his workflow’s parallel work from roughly two or three tasks to more than 15 while he worked on a TypeScript-to-Rust rewrite.

He also claims the Rust-based type checker was substantially faster than TypeScript’s and the planned TypeScript Go rewrite on his real-world codebases.

Format and Sources

No step-by-step guide is provided. The video is primarily an analysis and comparison of model capabilities, benchmark results, usage, and training dynamics.

Theo is the main speaker. He refers to conversations with fellow YouTuber Fami and colleague Guillermo; benchmark work by Justin Lo using DeepSwi and Max Bench; Artificial Analysis rankings; usage data from T3 Code; OpenRouter; reporting from Orca Router; comments from Railway’s Jake; and charts or demonstrations associated with Dara. Anthropic and an external report by Meter are also discussed.

Rate this summary

Your feedback will help improve summaries.

Improve this summary

Reprocess with a stronger model when the summary feels incomplete or inaccurate.

Pro

Translate summary in another language

Pro

Ask questions to this video

Chat for follow-up questions, clarifications, and source-backed answers.

Coming soon

Share this summary

Original video