Anthropic just built it...

Summary

The video discusses an independent METR assessment of how much AI is accelerating Anthropic’s AI research and development, alongside changes to Anthropic’s IPO filing concerning recursive self-improvement (RSI). The presenter argues that the reports leave important questions unanswered because Anthropic redacted substantial information.

Main Findings and Concepts

• AI-assisted research is increasing: METR’s assessment examines whether AI tools are speeding up AI R&D. Anthropic employees interviewed for the review generally said AI—particularly Claude—helps them run more experiments and reduces the time between an idea and its implementation. One reported estimate was an 80% reduction in that interval.
• A proposed “2×” warning threshold: Anthropic’s responsible-scaling framework treats a doubling of AI-driven research progress relative to its historical rate as a potential warning point. The assessment’s central estimate is described as roughly 1.9×, but with substantial uncertainty—possibly around 1.3× to 3×. METR reportedly found no clear evidence in the published benchmark trends that the 2× threshold had been crossed.
• Benchmark limits and AECI: Anthropic uses a partly undisclosed benchmark suite called AECI, combining internal and external tests. The presenter explains that ordinary benchmarks become less informative as models approach near-perfect scores, a problem known as benchmark saturation. AECI appears to use increasingly difficult tests and a scale based on logarithmic odds to better track capability.
• New, unreleased models: The presenter describes Anthropic models called Model 2, Model 3, and Model 4, which were trained during 2026 and were not generally available. Model 4 is said to exceed 50% on all tests in the AECI suite, while earlier models scored lower on the hardest tests. This suggests the benchmark suite may no longer provide enough difficulty to measure the newest model precisely; it does not establish that the model has surpassed every AI benchmark.
• Possible acceleration beyond compute scaling: METR reportedly concluded that the observed acceleration was not simply due to more training compute. The presenter focuses instead on algorithmic improvements and more efficient experimentation, while noting that the publicly available evidence is incomplete.
• Redactions and compute allocation: Anthropic reportedly narrowed the scope of METR’s review during the audit, excluding verification of how compute was allocated. The presenter connects this change to extensive redactions concerning model-training details, inference compute, and the processes behind the acceleration. Anthropic co-founder Sam McCandlish is quoted as saying that disclosing the specific drivers could give competitors a roadmap.
• Agent framework hypothesis: The presenter speculates that an undisclosed agent framework, combined with a stronger model, may let large numbers of AI agents carry out research and engineering tasks, turning researchers’ ideas into experiments much faster. The video cites an Anthropic snapshot reporting about 30,000 agents used for research and engineering work. It also mentions a July compute snapshot in which 6% of compute reportedly went to AI R&D, and 12% of AI-driven R&D compute went to security. The presenter suggests compute allocations may have shifted later, but emphasizes that this is a hypothesis, not a confirmed explanation.
• RSI and trend uncertainty: Anthropic’s revised filing is presented as saying recursive self-improvement is inevitable or approaching. The presenter discusses a gradual progression from AI assisting research to leading parts of it, and notes Anthropic’s reported estimate that AI took a leading role in 26% of certain projects in August, up from 1% in March. The video cautions that benchmark trends alone do not prove hyperbolic growth, though METR reportedly says redacted evidence could make stronger acceleration more plausible.

Analysis and Caveats

The video’s central concern is that a major change may have occurred around October, but the public reports do not reveal enough to establish what it was. The presenter’s explanation—that a newly deployed agent framework is converting many parallel AI research efforts into faster algorithmic progress—is explicitly speculative. Redactions, changing audit scope, and benchmark saturation make it difficult to assess the pace and cause of progress confidently.

Main Speakers and Sources

• Wes Roth: Presenter and primary analyst.
• METR: Independent organization whose assessment of AI-driven R&D acceleration is discussed.
• Anthropic researchers and Sam McCandlish: Sources for employee interview findings and Anthropic’s explanation for redactions.
• Daniel Kokotajlo: Referenced for commentary on Anthropic’s RSI-related filing.
• Anthropic reports and filings: Sources for the responsible-scaling threshold, model-progress discussion, compute snapshot, and RSI claims.