Video summary

Forget RAG Pipelines—Build Production Ready Agents in 15 Mins: Nina Lopatina, Rajiv Shah, Contextual

Main summary

Key takeaways

Technology

Summary (technological concepts, product features, and guidance)

  • “Managed RAG” positioning (avoid building pipelines from scratch): The talk argues you can treat RAG as a managed service—no need to train your own LLMs or build vector DBs yourself. Contextual provides an end-to-end platform that handles major RAG components, including: document ingestion/understanding, chunking, retrieval, reranking, grounded generation, and evaluation.

  • Why production RAG is hard: Demos often work with ~10 documents, but production challenges include:

    • scaling to thousands of diverse documents
    • accuracy and extraction quality
    • user querying difficulty (users often don’t know how to ask well)
    • operational overhead from coordinating many models (e.g., BM25 + embedding retrieval + rerankers + filters)

System architecture: core pipeline components

  1. Document understanding / extraction pipeline

    • Extracts content from PDFs, including tables and multimodal elements (images/figures).
    • Performs layout analysis, OCR when needed, and image captioning.
    • Uses document structure (section headers/hierarchy) as meaningful signal.
    • Produces chunking-ready representations with bounding boxes / attribution mapping.

  2. Chunking

    • Breaks documents into smaller pieces so the model doesn’t require long contexts.
    • Injects hierarchy/metadata into chunks.

  3. Retrieval (hybrid)

    • Uses a combination of lexical (e.g., BM25) and semantic (embeddings) retrieval.
    • Supports data store filters to restrict retrieval to relevant subsets (e.g., Apple vs other companies).
    • Defaults mention retrieving many chunks first (example: ~100) and then refining.

  4. Reranking

    • Uses a trained instruction-following reranker to improve accuracy.
    • Typically reduces the retrieved set from ~100 to a smaller top-K (e.g., ~15; adjustable).
    • The reranker can accept instructions/prompts (e.g., “prefer the most recent documents”).

  5. Grounded generation

    • Uses Contextual’s grounded language model (not OpenAI/Gemini per the talk).
    • Goal: keep answers grounded in provided context rather than relying on prior model knowledge.
    • Supports separating facts vs commentary (option to output only facts).

  6. Groundness + attribution checks

    • Attribution UI: answers reference supporting document regions/chunks (e.g., clickable sources).
    • Groundness check: response is decomposed into claims, and each claim is checked against retrieved context.
      • Claims not found are highlighted (e.g., yellow) to indicate potential hallucinations.

Default UX + adjustable “knobs”

  • The platform offers:

    • No-code/opinionated defaults for business users (upload docs and ask questions).
    • Developer controls for pipeline orchestration and tuning, such as: query reformulation/expansion, retrieval settings, reranker settings, filters, prompt/system prompt, suggested queries, etc.

  • They disabled multi-turn during the workshop for consistency, but multi-turn is implied as configurable.

Step-by-step tutorial/workshop flow (hands-on)

  • Get started via a notebook and setup page (contextual.ai25) to obtain an API key.
  • Create a workspace (unique name requirement mentioned).
  • Use the notebook to:
    1. install dependencies/imports (pip installs)
    2. create a data store
    3. download and ingest example documents (NVIDIA financial statements + “spurious correlations” files)
    4. inspect parsed outputs (raw text + structured previews; tables show structure and exact values)
    5. create an agent and ask questions in both:
      • the GUI (app.contextual.ai → Agents)
      • the API (programmatic querying)
  • Demo query types:
    • multi-document numerical aggregation (summing quarterly revenue across fiscal years)
    • time-based comparison (when data-center revenue overtook gaming revenue)
    • spurious correlation reasoning using provided documents:
      • the agent provides the correlation coefficient and caveats from the source documents
      • compared against:
        • “no-RAG ChatGPT” (refuses/doesn’t find correlation)
        • “RAG ChatGPT long-context” (may mention correlation but often won’t explain caveats thoroughly)

Evaluation guidance: LMU unit testing

  • Introduces LMU unit: an endpoint acting as a fine-tuned judge for natural-language unit tests.
  • Evaluation method:
    • Write unit tests (specific questions/criteria) for expected behaviors.
    • Run the RAG system to generate responses.
    • Score unit tests on a 1–5 scale using the LMU unit judge model.
  • Example unit test categories used in the notebook:
    • correct numerical extraction
    • correlation vs causation distinction
    • multi-document calculations
    • acknowledgment of limitations/uncertainties
    • evidence-backed quantitative claims
    • avoiding unnecessary information (verbosity control)
  • Results are visualized (including polar plots) and mapped to categories like accuracy, evidence, relevance, limitations, synthesis, etc.
  • Suggested “homework”: use unit test results to improve the agent, e.g., updating the system prompt (e.g., “keep responses brief, only answer the direct question”).

Modular subcomponents + standalone usage

  • Emphasizes Contextual can be used as:
    • an end-to-end platform, or
    • individual components (parser, reranker, grounded LM, LMU unit)
  • Components are accessible via APIs and SDKs (Python/JS mentioned).

Integration with MCP (tooling ecosystem)

  • Bonus demo: connect the Contextual RAG agent into other MCP clients (example shown with Claude Desktop / “Claude” and also mentions Cursor).
  • Workflow:
    • clone a contextual MCP server repo
    • host server locally
    • configure the client to point to the MCP server
    • client calls the Contextual agent for answers based on its tool description
  • Also mentions they provide GitHub directions and a how-to video.

Deployment, governance, and scalability Q&A highlights

  • Where it runs / data residency controls

    • Runs as SaaS; can also run in customer VPC.
    • Mentions partnership with Snowflake.
    • Not yet supporting AWS GovCloud specifically (at least at time of talk), but willing to discuss.

  • Scaling

    • Claims they handle customers with tens of thousands of documents (e.g., Qualcomm).
    • Mentions platform engineering support for scaling concerns.

  • Permissions / entitlements

    • Introduces need for an entitlements layer to enforce document-level access controls (permissions).

  • Frequently updated content

    • Mentions a continuous ingestion pipeline.

Cost/pricing model

  • Component-based pricing is consumption-based (token-based) for: parser, reranker, generation, and unit tests.

  • Platform pricing is also consumption-based: based on documents ingested and queries.

  • Workshop started users with $25 credit; exact limits weren’t quantified, but they encouraged trying it.

  • Mentions potential provisioned throughput for enterprise latency/throughput guarantees.

Main speakers / sources

Speakers

  • Rajiv Shah — Chief Evangelist, Contextual AI
  • Nina Lopatina — Presenter; NLP/language modeling background
  • Additional team members mentioned:
    • Matthew (platform engineer; scaling/backend questions)
    • John (solution architect; integration/environment questions)

Product sources referenced

  • Contextual AI platform + notebooks + API/MCP server repo
  • Models/endpoints referenced:
    • instruction-following reranker
    • grounded language model
    • LMU unit (natural language unit testing judge model)

Original video