Retrievers in LangChain | Generative AI using LangChain | Video 13 | CampusX

Overview

This video is a tutorial in the LangChain playlist focused on Retrievers—a core component for building RAG (Retrieval-Augmented Generation) systems.

Why retrievers matter in RAG

• A retriever is a component/function that takes a user query and fetches relevant documents from a data source (e.g., a vector store, Wikipedia API, websites/APIs).
• In LangChain, retrievers return LangChain Document objects (containing content + metadata).
• LangChain highlights that there are many retriever types for different use cases (not “one retriever”).
• All retrievers covered are Runnables, meaning they can be invoked and plugged into chains like other LangChain components.

How the retriever concept is explained

• System flow: User query → Retriever → search data source → select relevant documents → return documents
• The presenter uses an analogy of a “search engine” that outputs multiple documents.

Two major ways retrievers can be categorized

1. By data source (what they search)

   • Examples mentioned:
     • Wikipedia Retriever
     • Vector Store Retriever
     • Archive Retriever

2. By search strategy / mechanism (how they search)

   • Examples mentioned:
     • MMR
     • Multi Query Retriever
     • Contextual Compression Retriever

Retriever types taught (with Google Colab code demos)

The speaker says all examples are implemented in a Colab notebook and encourages trying/modifying the code.

1) Wikipedia Retriever (data-source-based)

• Uses the Wikipedia API to fetch articles relevant to a query.
• Relevance is based on keyword matching (not “syntactic search”).
• Demo details:
  • Create WikipediaRetriever with parameters like top results and language.
  • Call .invoke(query) to demonstrate that the retriever is a Runnable.
  • Print retrieved Document objects and extract page_content.

Key clarification: Even if it “loads” Wikipedia content, it’s still a retriever because it searches/selects relevant articles rather than loading everything.

2) Vector Store Retriever (most common retriever type)

• Fetches documents from a vector store using semantic similarity via embeddings.
• Demo details:
  • Build a Chroma vector store from sample documents using OpenAI embeddings.
  • Create a retriever from the vector store (example uses K/top results).
  • Call .invoke(query) to retrieve the most similar documents.
  • Also contrasts it with calling vector store methods directly (e.g., similarity search).

Key analysis: Why use a retriever if the vector store already supports similarity search?

• Vector store similarity search is usually limited to a single strategy.
• Retrievers provide an integration point and make it easier to use advanced strategies later.
• Benefit emphasized: retrievers are chain-compatible Runnables, enabling advanced retrieval behaviors.

3) MMR Retrievers (Maximum Marginal Relevance) — strategy-based

• Addresses redundancy: normal similarity search can return highly similar results.
• MMR aims to return results that are:
  • relevant to the query, and
  • diverse / non-redundant compared to each other.
• Demo details:
  • Use vectorstore.as_retriever(search_type="mmr", search_kwargs={...}).
  • Uses a lambda parameter (0 to 1):
    • lambda = 1 → closer to normal similarity search (more similar results)
    • lambda = 0 → emphasizes diversity

Example explanation:

• The climate-change example suggests MMR should prefer varied perspectives (e.g., glaciers vs wildfires vs coastal impacts).

4) Multi Query Retriever — strategy-based

• Solves query ambiguity.
• Approach:
  • Use an LLM to generate multiple query variants from the user query.
  • Run the retriever for each variant.
  • Merge results, deduplicate, and return the top results.
• Demo details:
  • Compare a vanilla similarity retriever vs the Multi Query Retriever.
  • Ambiguous example: “How to Improve Energy Levels and Maintain Balance”
  • Vanilla retrieval may get distracted by terms like “energy” / “balance” and return off-topic documents (e.g., “solar system”).
  • Multi Query generates variants more aligned to health/nutrition, leading to more relevant results.

5) Contextual Compression Retriever — strategy-based

• Improves retrieval quality by compressing documents after retrieval.
• Problem it targets:
  • Retrieved documents can contain multiple topics, so sending full documents can waste context and introduce irrelevant information.
• Approach:
  1. Use a base retriever (e.g., similarity search) to fetch candidate documents.
  2. Apply an LLM-based compressor that trims each document to only query-relevant parts.
  3. Return compressed (relevant excerpt) documents instead of full text.
• Demo details:
  • Input documents include unrelated topics in one paragraph (e.g., Grand Canyon + photosynthesis + basketball + cinema extras).
  • Query: “what is photosynthesis?”
  • Output becomes short, returning only the photosynthesis-relevant sentence(s) and removing unrelated content.

Other retrievers mentioned (not fully taught)

The presenter references additional LangChain retriever options, including:

• Parent Document Retriever
• Time Weighted Vector Retriever
• Self Query Retriever
• Ensemble / Multi retriever
• OnSumable Retriever
• General retrievers documentation (with links to sample code and descriptions)

High-level guidance / motivation

• Many retrievers exist because different retrievers solve different retrieval problems.
• For advanced RAG, the idea is often to swap in more advanced retrievers when a basic RAG system underperforms due to weak retrieval.

Main speaker / source

• Nitesh (presenter; “CampusX” video)