RAG Agents in Prod: 10 Lessons We Learned — Douwe Kiela, creator of RAG

Summary: RAG Agents in Production — 10 Lessons (Douwe Kiela)

Douwe Kiela (CEO at Contextual AI) explains why enterprise GenAI projects often fail to deliver ROI, and how RAG agents can be designed for production value rather than just demos. The core framing is a “context paradox”: language models are powerful but struggle to reliably apply the right context, while humans do context-setting easily—especially specialists.

1) ROI failure is about context + enterprise reality

• Enterprises face a paradox: LLM capabilities look impressive, yet only a minority of businesses report AI value.
• The “context” problem grows as you move from convenient help → business transformation goals.

2) Treat RAG as a system, not “the model”

• Lesson: LLMs are often only ~20% of a larger production system.
• A mediocre model + strong RAG pipeline (retrieval, grounding, processing) can outperform a top model with a weak RAG system.
• Recommendation: focus engineering effort on the full RAG system architecture.

3) Enterprise expertise is the real fuel—use specialization

• Enterprises have valuable internal knowledge, but general-purpose assistants struggle to match that expertise.
• Approach: specialize over “AGI”—build domain-focused solutions to better capture institutional expertise.

4) Data (not people) is the long-term enterprise asset

• Companies are defined by their data over time.
• Key challenge: enterprises think they must heavily scrub/clean data first; instead, the goal is enabling AI to work effectively on noisy data at scale.

5) Pilots are easy; production at scale is the hard part

• Building a RAG demo with a framework is relatively straightforward (small doc sets, limited users).
• Production requires:
  • Scaling to tens of thousands to millions of documents
  • Handling many use cases (potentially 20k+)
  • Meeting security + compliance requirements
• Strong guidance: design for production from day one, not pilot-first.

6) Speed beats perfection (iteration is essential)

• Production rollouts of RAG agents emphasize getting something usable early with real users, not testers.
• “Barely functional” first, then iteratively improve—waiting for perfection makes pilot→production transition difficult.

7) Keep engineers focused on business value, not low-level tuning

• If teams optimize for speed, they shouldn’t spend too much time on “boring” details like:
  • Optimal chunking strategy (varies by use case/framework)
  • Prompt micro-optimizations
• Recommendation: rely on modern platforms/abstractions where possible so engineers focus on differentiated business outcomes.

8) Make GenAI easy to consume (usage is often near zero)

• Many enterprises deploy GenAI but find nobody uses it.
• Reasons include workflow integration gaps and internal risk approval bottlenecks.
• Recommendation: integrate AI into existing enterprise workflows to drive adoption.

9) Build “stickiness” via fast “wow” moments

• Adoption improves when users get value quickly (onboarding/UX that triggers immediate insight).
• Example: Qualcomm users discovered long-hidden technical documentation through the system, rapidly resolving questions and creating strong perceived value.

10) Accuracy is table stakes—observability and attribution matter more

• Enterprises care about handling the missing 5–10% (what’s wrong or omitted).
• Recommendation: implement observability, including:
  • Proper audit trails
  • Attribution/grounding (“why the system answered” based on retrieved sources)
  • Claim checking / postprocessing to support evidence

---

Closing message: be ambitious, not incremental

Projects fail more often by aiming too low (small “basic Q&A” tasks) than too high. Aim for ambitious business transformations that actually generate ROI—“gimmicks” that don’t get used don’t achieve enterprise value.

---

Main speaker/source

• Douwe Kiela — CEO, Contextual AI (creator of RAG; previously at Facebook)