RAG is Dead: Jeff Huber (Chroma) on Building Agentic Search

Summary of Technological Concepts & Key Claims

1) “RAG is dead” (terminology + practical motivation)

• Jeff Huber (Chroma co-founder) argues the term “RAG” is vague: if you ask multiple people to define it, you’ll get inconsistent definitions.
• He also proposes banning the phrase “vector database”, suggesting teams should instead focus on what actually matters: context management.
• Core point: many AI agent failures today are context failures, not reasoning failures.
  • Too much context → “dumb zone” / “context rot”
  • Too little context → missing critical info the model can’t reliably infer
  • Models struggle with conflicting information and prioritization (which facts win)

2) Evidence: context rot / token-count non-invariance

• Chroma previously published a technical report (“context rot”) claiming LLM performance is not invariant to context window size.
• As more tokens are added, the model’s ability to reason well and attend to large portions degrades precipitously.
• Builders’ heuristics for “dumb zone” vary (e.g., 20k–120k tokens), and he argues it’s not trustworthy to assume millions of tokens “just work.”

3) “Context engineering” should shift from hand-built pipelines to model-driven behavior

• Instead of heavy manual context engineering, he suggests tooling + guidance, letting agents decide how/when to use tools.
• He claims context engineering will increasingly be folded into models themselves.
• Example direction: a Chroma-released model that can edit its own context (lightweight self-context engineering).

4) Tooling approach: avoid naive file-system-only “agent context”

• He discusses an agent pattern popular online: give an agent access to a sandbox/local file system + bash-like tools.
• He critiques file systems as “bad databases,” citing:
  • poor concurrency (e.g., last write wins)
  • risk of corruption in pathological cases
  • no indexing over content
  • limited search (e.g., grep/pattern matching only)
  • sandboxes may be too heavyweight if you only need read/write access

5) Alternative: virtualized file systems (lightweight context layer)

• A more scalable approach is gaining traction:
  • provide agents a virtualized bash / virtual file system that looks real to the agent
  • agent commands like grep, cat, ls, etc. work
• Example cited: Mintlify’s article using Chroma as backing storage behind a virtual file system.
• Framed as a lightweight way to provide agent context without heavyweight sandbox execution, especially for read-only cases.

6) Rethinking grep + recommending hybrid / genetic search defaults

• If an agent uses grep-like workflows, the system doesn’t need to restrict results to literal grep regex—results can be padded/expanded with extra context.
• He downplays the idea that “agents should use file systems because agents are trained on FS APIs” as a short-sighted meme.
• Default recommendation for finding information:
  • “genetic search with hybrid search under the hood”
  • positioned as a “magic button” baseline for many use cases

7) Chroma product releases highlighted

• Chroma Sync
  • drop files into a bucket
  • Chroma automatically handles indexing (avoids building custom pipelines)
• Chroma Context One (open-source model)
  • a specialized search sub-agent
  • takes a natural-language query but outputs documents only
  • those documents are passed to a separate “reasoning/generation” model for the final response
  • positioned to avoid “gigabrained” search (expensive/slow top-tier reasoning models)
  • claims: trained with a small budget/team, reporting 10x faster, 25x cheaper, and better or on par with baseline search on public/private benchmarks

8) Q&A insights captured in subtitles

• Memory vs context compaction
  • Chroma-style primitives can support memory, but memory is essentially context distillation / context compaction: selecting what to inject from a large memory pool into a small prompt.
• Storage integration question (Firebase)
  • their approach can work with other storage solutions (mentions Firebase as something the audience can integrate/layer on).
• Is agentic search replacing RAG fully?
  • he avoids using “rag” and emphasizes using data and running evaluations.
  • Empirically:
    • On small corpora (e.g., 30–50 files), simple grep eventually finds the right thing (may be slow/expensive).
    • On larger corpora, grep breaks down.
    • If you need semantic expansion (not knowing exact query terms), you need more expressive search.
  • he says they will release evaluation numbers for a clearer answer.

---

Main Speaker / Sources

• Main speaker: Jeff Huber (co-founder of Chroma)
• Referenced sources / products / tools:
  • Chroma technical report: “context rot”
  • Swyx article: “Oops, You Wrote a Database”
  • Mintlify (virtual file system approach; Chroma used as backing DB)
  • Mentions of evaluations/inflection discussions tied to “Cloud Code” (via Boris Chen)
  • Product/model releases: Chroma Sync, Chroma Context One (open-source on Hugging Face)