Video summary

Jev Creator on Will US Open Models Catch Chinese | The Cost of Intelligence & Tokens | Diogo Almeida

Main summary

Key takeaways

Technology

Summary

Diogo Almeida, co-founder of TypeSafe AI and creator of Jev, argues that AI reliability—not raw capability—is the main obstacle to automating real work. He distinguishes intellectual reliability from uptime or availability: a model may be brilliant most of the time yet make occasional, unpredictable mistakes. He illustrates this as “Einstein 95% of the time and Mr. Bean the other 5%,” saying that such inconsistency makes models hard to trust with important decisions.

Jev and Reliable Automation

Almeida presents Jev as focused on making AI dependable for background automation and decision-making, rather than merely helping a human user or achieving strong benchmark scores.

He says reliability depends on improving judgment and decision quality. In his view, failures may come from the model, the prompt, or the surrounding software design, so developers need to diagnose and address all three.

He believes AI could eventually enable software to automate tasks that current applications leave to people. That requires substantial engineering around models, not just adding a chatbot or copilot to existing products.

What Makes Models Improve

Almeida gives this priority order: the right task, the data for that task, compute, then algorithms.

  • Task-specific data: Data is valuable only when it is designed for a clearly defined objective. New tasks often require creating new kinds of data; for example, useful human preference data for RLHF did not already exist.
  • Compute and scaling: More compute can still help, but scaling alone requires more data than the industry initially expected. Almeida says scaling became less economically compelling when improvements did not justify the cost.
  • Synthetic data: He describes synthetic data as a tool rather than a universal solution. Its usefulness depends on how it is applied and whether it advances the target capability. Jev uses synthetic data with a narrow focus on robustness.
  • Access to capable models: Data quality depends partly on whether it can improve strong models, making access to capable models important for data-development work.

Benchmarks, Model Comparisons, and Open Models

Almeida is highly skeptical of public benchmarks. He says they can reward optimization for test scores rather than useful, reliable behavior, potentially giving buyers a misleading picture of model quality. He argues that organizations should evaluate models on real tasks and failure cases instead.

On open models, he claims that some labs overemphasize benchmark results for organizational or competitive reasons. He expects U.S. open models may not match Chinese models in raw capability unless Chinese development slows for ethical reasons. However, he says the practical significance of a capability gap could diminish if models reach a useful plateau.

Almeida also notes that security concerns depend on the use case. Safeguards and systems that inspect tool calls can provide protections beyond choosing a particular model.

Software and Enterprise AI

Almeida says most enterprise software remains largely unchanged because unreliable models are not suitable for consequential decisions. As a result, AI is often added as a side feature—typically a user-facing assistant—while people remain responsible for checking its output.

He sees established software companies as well positioned not simply because of their distribution, but because they understand existing workflows and can invest in the engineering needed to automate them. He favors automating specific tasks that can be made dependable over trying to create a general-purpose AI “colleague.”

In his view, a company’s distinctive intelligence may come more from the software and engineering built around models than from training its own proprietary model. He is critical of copilots that only make workers somewhat more efficient when larger portions of a workflow could ultimately be automated.

He acknowledges that specialized models may offer cost or focus advantages, and that their value depends on the task and the point of diminishing returns for larger models.

Token Demand, Efficiency, and the AI Market

Almeida predicts that dependable decision models could lead to a large increase in AI use. If AI can reliably automate work in the background, it may be used in many more places than conversational LLMs. He frames this as a large Jevons-style effect, in which greater efficiency and usefulness drive more overall consumption. Achieving this would require further efficiency improvements and more work with developers to make automation practical.

He is less convinced that general intelligence per token is improving rapidly. He sees substantial gains in coding as partly resulting from models being better adapted to coding tasks, rather than becoming uniformly more capable across all domains.

Almeida also expresses concern that AI investment may be in a bubble. He thinks demand could weaken if companies overestimate what current models can do and fail to realize the expected value. At the same time, he argues that reliable automation could create new demand by making work automatable that AI does not currently handle.

Reviews, Guides, and Tutorials

This is an interview and analysis discussion. It does not provide a step-by-step tutorial or a hands-on product review.

Main Speakers and Sources

  • Diogo Almeida — TypeSafe AI co-founder and creator of Jev; provides the principal views on reliability, data, model evaluation, automation, and the AI market.
  • Harry Stebbings — Host of 20VC; asks questions and challenges Almeida’s arguments.

Rate this summary

Your feedback will help improve summaries.

Improve this summary

Reprocess with a stronger model when the summary feels incomplete or inaccurate.

Pro

Translate summary in another language

Pro

Ask questions to this video

Chat for follow-up questions, clarifications, and source-backed answers.

Coming soon

Share this summary

Original video