Video summary

We Tested Every AI on Research Math

Main summary

Key takeaways

Technology

Rapid Progress in LLM Math Reasoning

The video claims that state-of-the-art large language models have significantly improved over the past year on research-level math problems, including cases that earlier models couldn’t solve.

Multi-Agent / Extended Thinking Tests (“Clankers”)

  • Models were tested in scenarios where agents “think in groups” and attempt to communicate about research projects.
  • One test involved an agent thinking for over 7 days, then reporting a solution.
    • The example mentioned deriving/using two inequalities with random numbers.

Core Experiment: Single vs Combined Model Pipelines

The presenter evaluates whether combining or routing-to-best model approaches produce better answers than relying on a single strong model (noted as GPT Pro previously being the best).

Tool + Workflow Introduced: Abacus AI

Abacus AI is presented as a subscription platform that provides:

  • Access to many models through one chat interface, including:
    • ChatGPT, Claude, Grok, Gemini
    • Chinese models such as “Kimmy” and “Deepseek”
  • The ability to scroll/select models per question.
  • A routing option (via a “root LLM” / router) that sends a question to the model believed to be most capable.
  • Automatic pre-formatting for technical/LaTeX-style answers inside the UI, reducing hassle versus CLI/API workflows.
  • Output aggregation/compilation into PDF/LaTeX documents.

Credit / Token Accounting (Example)

The presentation emphasizes credit usage, e.g.:

  • ~20,000 credits for $10/month

Math Domain Focus: Determinant → Advanced Theory

The specific problem centers on determinants, interpreted as the volume spanned by vectors.

The longer-term goal is to connect the determinant to:

  • Orthogonal polynomials
  • The associated Riemann–Hilbert problem

The presenter wants to see whether models can reliably make the “hard jump” from direct determinant computation to the representation that makes the computation easier.

Per-Model Evaluation (One Determinant / Riemann–Hilbert-Oriented Problem)

Root LLM (Router) → ChatGPT / GPT5.6-like Model

  • The routed answer is described as best overall.
  • It includes the “nice formula” the presenter expected.
  • It also adds additional advanced context.

Deepseek V4 Pro

  • Early results looked strong.
  • Later, it hallucinated/misread the problem.
  • It reportedly mentioned unexpected Barnes G functions.
  • Uses 93 credits.

Kimmy K3

  • Uses credits efficiently (179 credits).
  • Provided a “standard theory” approach.
  • Included a numerical verification attempt.
  • Had formatting bugs.

Grok

  • Described as “standard.”
  • Used 331 tokens.
  • Did not cite the expected “nice formula.”

Gemini Pro

  • 74 credits.
  • Quick response with little/no chain-of-thought.
  • Surprisingly token-efficient and comparable in quality.
  • Still did not reach the “nice formula” depth.

Routing Impact and PDF Compilation

For PDF generation, the reported pipeline:

  • Started from Markdown outputs produced by models.
  • Used the router (Root LLM) for initial model selection.
  • Used another model (described in the subtitle as “GPT5.6… terror”) to compile everything into a single summarized PDF/LaTeX document.

Outcome: a combined, LaTeX-looking summary that merged multiple model strategies into one research document.

Conclusion / Recommendations

The presenter is impressed by Abacus’s:

  • Low-friction, multi-model access
  • Routing-to-best behavior
  • Technical formatting
  • Document aggregation

They argue it helps with researching which model works best per domain and per question.

Main Speakers / Sources Mentioned

  • The video’s presenter (primary speaker; described as a PhD researcher using PhD-level problems)
  • Abacus AI (platform sponsor/tool source)

LLM Providers / Models Used for Comparison

  • ChatGPT (GPT5.6)
  • Claude
  • Grok
  • Gemini
  • Deepseek V4 Pro
  • Kimmy K3

Original video