RAG is Dead - Introduction to Vectorless RAG

Summary

The video introduces vectorless RAG, also called PageIndex, and contrasts it with traditional retrieval-augmented generation (RAG). Despite the title, the presenter notes that vector-based RAG is widely used. Vectorless RAG is presented as an alternative, not a universal replacement.

Traditional RAG

Sending an entire document to an LLM for every question can exceed its context window, reduce answer focus, and increase token costs. Traditional RAG addresses this by:

1. Splitting documents into chunks.
2. Creating vector embeddings for those chunks and storing them in a vector database.
3. Embedding the user’s query and retrieving the most similar chunks.
4. Providing the retrieved chunks and query to an LLM to generate an answer.

The presenter highlights several shortcomings:

• Chunk boundaries can split related information.
• Fixed-size or paragraph-based chunking may not reflect a document’s actual structure.
• Vector similarity can miss relevant material when the user’s wording differs from the source.
• Cross-references, such as a legal clause pointing to another section, may require retrieving information from multiple places.

Vectorless RAG / PageIndex

PageIndex does not rely on vector embeddings or similarity search. Instead, an LLM analyzes a document and builds a hierarchical table-of-contents tree. Nodes can contain titles, summaries, child nodes, and IDs or location references that point back to the original document.

At query time, the system uses LLM reasoning to navigate the tree, select relevant nodes, and retrieve the corresponding source material for answer generation. The presenter compares this to using a book’s contents to find the relevant chapter rather than searching every page indiscriminately.

The Sholay movie example illustrates how the tree might organize a story by scenes, characters, and important events. The presenter also explains how structural navigation may help with documents whose relevant information is connected across sections.

Tradeoffs and guidance

• Potential benefit: Retrieval is based on document structure and reasoning rather than fixed chunks and vector similarity.
• Tradeoffs: Reasoning models can increase cost and latency. The approach trades speed for potentially more accurate retrieval.
• Implementation overview: The video explains the pipeline conceptually and mentions a Python PageIndex repository, but does not provide a coding tutorial or comparative benchmark.

Main speaker and sources

• Main speaker: Piyush Garg.
• Sources and technologies mentioned: PageIndex materials and repository; OpenAI embedding models; and vector databases including Pinecone, Chroma, Weaviate, Milvus, Qdrant, and PostgreSQL with pgvector.