Video summary

It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners

Main summary

Key takeaways

Technology

Technological summary (RLM / recursive language models)

What RLM is

A recursive language model (RLM) differs from typical tool-calling and agent patterns by treating context as an object in a symbolic execution environment—often a REPL (e.g., a Python REPL). Instead of only sending strings/JSON and receiving strings back, the model symbolically interacts with that environment.

Key mechanism: delegation inside the REPL

  • RLMs can delegate / sub-call to other LLMs (often itself, sometimes different models) within the same REPL-like environment.
  • This supports recursive decomposition of problems, where subordinate LLMs can further delegate.

Why it helps with long-context and “context degradation”

Common systems struggle as the context window fills (performance drops). RLMs mitigate this by:

  • Avoiding stuffing everything into the context window at once.
  • Keeping information as variables in the REPL.
  • Letting submodels receive only what’s relevant.
  • Deciding how to access/use context, returning only meaningful intermediate results.

Performance / benchmarks mentioned

The speaker references long-context benchmarks, including:

  • ULong: answering questions over very long contexts.
  • BrowseComp: searching through a large corpus/array to answer targeted questions.

Reported takeaway: RLM (blue line) outperforms other approaches on the mentioned benchmarks. In one comparison, a baseline using GPT tool calls + BM25 is described as more expensive and worse-performing than RLM.

Contrast vs RAG, agents, and tool-calling

  • RAG often “fills” the context window quickly and hits limits.
  • Agents / tool calling frequently return strings, creating weaker coupling between:
    • logic,
    • execution,
    • structured intermediate results.
  • RLM keeps intermediate artifacts as script / REPL variables, enabling tighter integration between computation and results.
  • The speaker notes Anthropic’s workflows as an emerging similarity: intermediate results stored in variables (mentioned via a conference/paper).

Product / features & “how to use it” patterns

Mental model / programming abstraction

The speaker emphasizes a higher-level abstraction: define intent + input/output types, and the RLM system handles internal implementation details. They compare this to DSPy, described as a deterministic-ish “shell” with structured input/output guarantees—suggesting similar adoption benefits.

When RLM is a good fit (explicit use cases)

RLM is framed as especially useful for:

  • Large/dense contexts (long documents, long traces, complex corpora).
  • Source/data extraction or generation workloads, including producing hundreds of thousands of rows.
  • Decomposable research tasks, e.g., iterative analysis across sections of something like a tax code.
  • Long-horizon sessions that benefit from iterative sub-analysis.
  • Cases where you want to avoid heavy manual context engineering and rely instead on structured REPL integration.

When to skip / limitations

  • If the task fits easily into context, or you need low latency and the approach becomes too “programmer-heavy,” RLM may be less suitable.
  • The speaker’s framing: best for tasks that are structured, decomposable, and context-heavy, not always for simple one-shot prompts.

Code execution / data-frame interaction example

An example is described for cohort retention analysis using dataframes in the REPL:

  • Provide multiple dataframes and instructions about which raw data types to attend to.
  • RLM generates code, iterates by directly interacting with the dataframe (similar to a notebook), and returns structured outputs.

The speaker highlights:

  • Tracing and separation of:
    • generated code,
    • intermediate reasoning steps,
    • final formatted key findings + recommendations,
    • a terminal “submit” step.
  • An explicit stop condition: “LLM decides when to stop,” bounded by a max iterations parameter.

Reported improvements / experimental claims

“Long chain-of-thought” / accuracy jump

The speaker cites performance testing showing improvement from 2.6% to 45.4% accuracy on some “long chain of thought” style tasks (link referenced but not included in the subtitles).

Task types where RLM shines

Examples include:

  • Logic puzzles,
  • chess,
  • chemistry,

Where it can write code, extract relevant context portions, compute results, and aggregate outputs from sub-LLMs.

Examples about token-availability limits

For problems hidden inside huge token streams (e.g., “adding 12 numbers hidden among 30,000 tokens”), the speaker claims:

  • a basic LLM may struggle,
  • while RLM can solve it via code / regex-like structured computation.

A second example discusses dataframe-related experiments where tool-call based approaches (string/JSON tool calling) are less efficient than direct REPL dataframe interaction.

Libraries / platforms / projects mentioned

  • Predict RLM: focused on RLM; works with spreadsheets/PDFs.
  • DSPy: not purely RLM, but integrated/compatible; used as a structured orchestration layer.
  • Ax: mentioned as interesting RLM-adjacent work.
  • Faster RLM and an inference.net-related tweet:
    • Uses RLM to extract insights from work traces to decide what should transfer to newer models (mentioned alongside GLM 5.2 context).
  • Trampoline AI:
    • Uses RLM for intellectual work over PDFs and spreadsheets.
    • Example: consolidating many messy invoices into one list without manually handling 200-page context/chunking/embedding strategies.
    • Claim: uses DSPy to learn patterns between calls to the main LM and auxiliary LMs, improving readability/maintainability and enforcing structured return types.
  • Halo:
    • Uses RLM to analyze agent traces and iterates the “harness”/framework itself—described as meta-optimization.
  • AWS engineer experimentation:
    • Uses RLM on large logs/traces to uncover insights.
  • OWASP intentionally vulnerable app example:
    • Runs an agent over a large codebase (e.g., 500,000 lines) to generate a security report, emphasizing minimal context engineering.

Key “bottom line” message

The speaker argues you shouldn’t over-focus on manual context engineering. With RLM, you can:

  • define goals and structured I/O,
  • let the model exploit REPL structure,
  • use symbolic context variables,
  • rely on recursive delegation,

…to reduce cost/complexity/bloat and improve performance on long/context-heavy tasks.

Main speakers / sources

  • Kevin Madera (Kevin Madera) — from Alex Partners (speaker)

References mentioned in the talk

  • Omar — advisor on RLM issues at Alex; source of an early tweet/concept
  • Raymond — mentioned for performance testing (“long chain of thought” test)
  • Sam Hogan — runs inference.net
  • Tariq — mentioned in relation to Anthropic workflows / a paper discussion
  • Dex — coined “dumb zone” in a contextual engineering talk
  • AWS engineer — experiment referenced
  • OWASP — example application source

Original video