We Gave Fable 5.1 an Unsolved Math Problem...

Main ideas / lessons

• New model release skepticism: The video argues that benchmark-heavy claims by AI providers may not reflect real-world performance on research-level math.
• Test methodology: Instead of using vendor benchmarks, the creator tests Anthropic’s Fable 5.1 on a set of hard math problems drawn from their own PhD thesis problems.
• Real-time comparison approach: The creator compares performance using the same general workflow they previously used with GPT-5.6, including exporting results and checking correctness via supporting computation.
• Artifacts matter: The creator values not only answers, but also:
  • the ability to do long multi-step work,
  • produce verifiable code/tests (Python),
  • and deliver readable summaries (PDF formatting) rather than only raw Markdown.
• Progress is uneven across tasks:
  • Fable 5.1 performs impressively on some tasks (including producing diagram-based comparisons and writing scripts).
  • For at least one more complex task, it cannot replace a “machine approach” with a purely analytical proof, and the attempt runs into technical/operation constraints.
• Practical “agentic” prompting: The video highlights a method for making a model “think for a long time” by giving it a structured folder/repository with instructions and stopping criteria—similar to running a batch job.
• Cost/token usage: The creator notes heavy usage of model tokens and sessions during the testing.

---

Methodology / step-by-step instructions (as described)

A) Test Fable 5.1 on real research problems (instead of relying on benchmarks)

• Collect a list of research-level math problems (in this case, problems the creator encountered during their PhD thesis).
• Organize them in a repository-style “folder” intended for advanced model tasks (the “research folder” / “Klein career research folder”).
• For each task:
  • Provide the problem with enough context and goals.
  • Request the model to produce outputs (often in Markdown) plus any computation it uses.
  • Ask for a PDF summary for readability when the Markdown is hard to read.

B) Use verifiable computation support

• When evaluating whether the model’s proposed solution is correct:
  • Inspect whether it produced Python scripts and tests.
  • Compare computed quantities (numerical experiments) to the targeted mathematical objects (e.g., density functions / statistics from random matrix theory).

C) “Make the model think longer” using a file-system/repository instruction set

• Create a folder/repository that contains:
  • problem statements,
  • what has already been proven,
  • which parts are open vs tested,
  • guidance on what the model should attempt next,
  • and a clear expectation to keep working until the folder’s tasks are completed.
• Provide this folder to the model so that:
  • it works as if solving a structured set of tasks,
  • it can update files and track progress,
  • it is not constrained to a single short chat turn.
• When outputs become hard to read in raw Markdown:
  • compile/export model outputs into PDFs for easier review.

D) Evaluate whether solutions are “analytical” vs “machine-based”

• After the model proposes results:
  • check whether it can replace a computational/mechanistic method with a human-style analytic proof.
• If the prompt becomes too complex:
  • the attempt may fail due to operational constraints or because no fully analytical strategy is found.

---

Main problem areas discussed

1) Random matrix theory / algebraic geometry flavored approach

• The first task relates to studying structure in higher dimensions starting from lower-dimensional cases.
• The model is credited with:
  • producing a solution framework,
  • creating diagrams comparing the target quantity to numerically computed approximations,
  • implementing Python-based validation,
  • and reducing a difficult expression to an algebraic function described via ratios of polynomials and square roots.
• The creator says the write-up includes advanced concepts such as:
  • layers and stratifications (linked to algebraic geometry),
  • plus explicit chains of polynomials.
• The creator finds the language shift (“speaks the language of mathematics”) encouraging and suggests a possible future direction: building a “rigid program” (bibliography / study plan) using algebraic geometry tools to understand why this language fits the problem.

2) Repository-based long-running research attempt (encoding/mode application context)

• A second task is framed as part of longer dissertation-follow-up work:
  • it references developing/continuing a formula with applications “like encoding modes.”
• The creator emphasizes setting up a research tracking repository so future agent runs can:
  • compare versions,
  • log proven vs open pieces,
  • and progressively solve problems from context without restarting chats.

3) Moving from known results to a more analytical proof

• In the long-running work, Fable 5.1 reportedly:
  • reproduces known results (the creator views some as potentially redundant),
  • provides formulas involving determinants and “boundary” mechanisms (and references Segal’s theorem by name in connection with fixed-end/fixed boundary approaches).
• The next harder task is described as:
  • GPT-5.6 produced “machine-like proof” content with specific inequalities,
  • and the new goal is to replace parts of that with a more real mathematical proof.
• The creator eventually reports that, for this hardest target:
  • technical issues occur (API errors, output token overflow),
  • and the model still can’t fully provide an analytical strategy that replaces the machine approach.

---

Overall conclusions expressed by the creator

• Fable 5.1 shows meaningful improvement on some complex math tasks—especially with:
  • diagrammatic comparison,
  • Python verification,
  • clearer PDF-structured reporting,
  • and ability to work through repository-based long tasks.
• However, for the hardest goal (replacing a machine approach with a clean analytic proof), it does not succeed within the described attempts.
• Despite limitations, the creator is impressed relative to other models (including comparisons to GPT-6, and noting “Astra” is excluded due to enterprise-only status).

---

Speakers / sources featured (identified)

• The video creator / narrator (PhD thesis author; not named in the subtitles)
• Anthropic (company releasing/claiming improvements to Fable 5.1)
• Anthropic member who provided access/support: Claude Pro Max for 6 months (unnamed)
• GPT-5.6 (used previously by the creator as a comparator model)
• Codex / “Codex” (used as a point of contrast; no individual speaker)
• Claude (as part of “Claude Pro Max” and “Claude response exceeded token limits”)
• Astra (mentioned as an excluded model, enterprise-only)
• Referenced math author(s):
  • Conry (mentioned regarding an original result about zeros on the critical line)
• Referenced theorem (not a person in the subtitles):
  • Sego’s theorem (spelled as heard; exact intended name unclear)