Video summary

The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

Main summary

Key takeaways

Technology

Technological concepts & main claims (agentic search + BM25)

Agentic search definition

Agentic search is search performed inside an agent loop, where the agent iteratively:

  • formulates queries,
  • retrieves evidence,
  • reads results,
  • and continues until it completes the task.

Core components of an agentic search system

  1. Capable model: can use tools and formulate queries.
  2. Harness: exposes retrieval/search to the model (e.g., tool calling or code mode / “Expose retrieval infrastructure”).
  3. Retrieval engine: performs efficient search over very large corpora, potentially up to billions of documents.

BM25 concept

BM25 (Best Match for “25”) is a classic lexical scoring function that scores relevance via term interactions between a query and a document.

  • Presenter emphasis: BM25 itself hasn’t changed fundamentally—it’s primarily a scoring primitive.

Why BM25 is “unreasonably effective” now

Key reasons discussed:

  • The “new user” (LLMs/agents) is more capable:

    • can use general knowledge (entities, dates, companies, etc.) to improve query usefulness
    • can type/ read faster
    • can reformulate queries and use richer query syntax/operators
    • can generate many queries (multi-step search trajectories)

  • Exact/lexical matching still matters for identifiers such as:

    • names, entities, ZIP codes, SKUs
    • embedding models may struggle to represent these reliably

  • Cost/operational simplicity:

    • BM25 is cheaper and easier to deploy than embedding-based retrieval
    • avoids heavy embedding inference and related infrastructure

  • Explainability to the model:

    • the agent can inspect retrieved snippets
    • can understand why results matched literal terms
    • can improve subsequent query reformulation

  • Claimed synergy:

    • BM25 + GPT works well because both align with literal match / term-evidence workflows.

Evaluation & benchmarks discussed (agentic vs classic IR evaluation)

Why traditional IR evaluation may be less relevant

The argument: agents do more than a single “one query → one ranked list” step.

  • Classical metrics like NDCG over a single query are described as “dead” or less aligned with agent behavior.

BrowseComp+ benchmark (deep research / QA style)

  • Described as a benchmark with ~830 riddle-like questions
  • Dataset scale: ~100k web documents
  • Harness/tooling:
    • the model receives a simple search tool
    • the tool returns snippets for a query string
  • End-to-end success criteria:
    • success compares whether the model/loop produces the exact golden reference answer

Key observation

  • If evidence is pre-inserted into context, accuracy is high (reasoning is not the bottleneck).
  • With retrieval enabled, accuracy drops because performance depends heavily on:
    • query formulation quality
    • retrieval quality

Search trajectories analysis

  • In the benchmark, each question can become a multi-step:
    • query → retrieve → read → update context → eventually answer
  • The speaker references a Hornet blog post analyzing query formulation behavior (e.g., GPT-5).
  • Comparison of query behavior:
    • Human logs (e.g., AOL) show shorter query patterns
    • LLM agents can generate longer, more operator-rich queries (e.g., site operators, phrase syntax), creating a different search workload.

Context window motivation (floppy disc analogy)

The talk argues retrieval is necessary because LLM context windows are limited, likened to storing only a small amount of data (compared to something like a “floppy disc”).

BM25 implementation and parameter sensitivity

  • Prompted question: “Which BM25 do you mean?”
  • BM25 has hyperparameters and implementation differences.
  • Claim: a baseline BM25 configuration used in BrowseComp+ research was “terrible.”
  • More recent work suggests earlier parameterization may not handle:
    • long-document / long-content scenarios well

Implication

  • Effectiveness depends on choosing/optimizing BM25 parameters and implementation, not just using “BM25” as a label.

Retrieval + “workspace expansion” / VFS-style agent environment

Referenced research (Waterloo / Jimmy Lin’s group)

  • Paper: “Scaling Direct Corpus Interaction via Dynamic Workspace Expansion”

Core concept

For web-scale agent search over billions of documents:

  • you cannot place everything directly into context
  • use BM25 as a baseline retriever to fetch relevant documents
  • then place retrieved content into an agent “workspace” organized like a file system / VFS

Progressive disclosure

The workspace enables stepwise reading:

  • show title + snippet first
  • allow the agent to decide whether to read more

Workflow/sandbox framing

The workspace supports sandbox/tool workflows similar to coding agents, including primitives resembling bash/tooling behavior.

Overall framing

  • this approach combines retrieval infrastructure + sandbox/workspace management for a fuller agentic experience.

Systems/performance claims (throughput/latency)

Efficiency comparison

  • Presenter showed a chart comparing Hornet vs other engines on comparable hardware and at web-document scale (~100M documents on a single node).
  • Claim: Hornet delivers better latency/throughput (described as more throughput for the same hardware/budget).
  • Clarification in Q&A snippet: axes were clarified as QPS vs latency.

Takeaways (explicitly listed)

  1. The new user (LLM agent) makes BM25 more powerful via general knowledge, faster typing/reading, and better query reformulation.
  2. BM25 choice matters: implementation and parameters differ; wrong configurations can hurt performance (including mismatches for long-document benchmarks).
  3. BM25 works well for agentic search because it is explainable and complements GPT through evidence that supports literal matching and query iteration.

Main speakers / sources

  • Jo Kristian Bergum — CEO of Hornet.dev (main speaker)
  • Edo — referenced for demonstrating “code mode” for exposing retrieval infrastructure (likely a demo/presenter collaborator)
  • Jimmy Lin’s group (University of Waterloo) — referenced paper: “Scaling Direct Corpus Interaction via Dynamic Workspace Expansion”
  • BrowseComp+ benchmark — cited benchmark paper (speaker describes it as published last year)
  • AOL query log — referenced as an example source of historical human query patterns

Original video