Text Splitters in LangChain | Generative AI using LangChain | Video 11 | CampusX

Overview

This video (CampusX) explains “Text Splitters” in LangChain as a core building block for RAG (Retrieval-Augmented Generation). It focuses on why chunking is needed and how to do it using LangChain.

---

What is text splitting?

• Text splitting = breaking large documents (articles/PDFs/books/pages) into smaller chunks that an LLM can handle.
• A text splitter is the component/code that performs this chunking.
• Key point: Never send very large text to an LLM at once—quality typically degrades.

---

Why it matters in LLM/RAG apps (3 main reasons)

1. Model context length limits

   • LLMs/embedding models have maximum input sizes (example given: 50,000 tokens/words).
   • Large PDFs may exceed limits (example given: >100,000 words), so splitting is required.

2. Better downstream task quality

   • Embeddings: embedding a huge text into one vector can miss semantic details; embedding separate chunks improves representation.
   • Semantic (vector) search: search quality improves when matching query embeddings against embeddings of meaningful chunks, rather than an entire large document.
   • Summarization: summarizing large documents can “drift” or include content not present; chunking improves faithfulness/quality.

3. Compute/resource efficiency

   • Smaller chunks reduce memory/storage needs and enable better parallel processing.

---

Four ways of text splitting covered

The tutorial covers four techniques:

1. Length-based (character/token count)
2. Text-structure-based (hierarchical splits: paragraphs/sentences/words/characters)
3. Document-format-based (code/Markdown-aware separators)
4. Semantic-meaning-based (breaks where topic meaning changes)

---

1) Length-based text splitting (CharacterTextSplitter)

Concept

• Simplest/fastest: decide a fixed chunk size (e.g., 100 characters).
• Chunks are created by traversing the text and cutting when the size limit is reached.

Pros

• Very simple to implement and fast.

Cons

• Ignores linguistic structure/grammar/meaning:
  • It may split mid-word or mid-sentence/paragraph, hurting embedding/search quality.

LangChain implementation details (as taught)

• Uses CharacterTextSplitter
• Parameters shown:
  • chunk_size (e.g., 100)
  • chunk_overlap (initially set to 0)
  • separator (left as empty/none so it splits by count)

Connecting loaders + splitters (PDF example)

• Uses a PDF loader to create one document per page, then applies splitting:
  • PDF loader → split documents with split_documents → chunk documents (each chunk has page_content).

Chunk overlap (important)

• Chunk overlap = number of characters shared between consecutive chunks.
• Benefit:
  • Helps preserve context lost when a chunk boundary cuts abruptly.
• Trade-off:
  • Larger overlap ⇒ more chunks ⇒ more compute.
• Rule of thumb given:
  • For text apps with chunk_size ~100, overlap ~10–20 is recommended.

---

2) Text-structure-based splitting (RecursiveCharacterTextSplitter)

Core idea

Recursive splitting that respects text hierarchy:

• Split by paragraph separators first
• If needed, split by sentence/line separators
• Then words
• Finally characters as a fallback

Goal: avoid abrupt mid-word breaks and keep coherent units.

How chunk_size affects behavior (demonstrated)

• Smaller chunk sizes push recursion deeper (more granular splitting).
• Larger chunk sizes allow bigger structural units (paragraph-level splitting).

Examples in the video:

• chunk_size=10 ⇒ more fine-grained splits (down to words/characters)
• chunk_size=25 ⇒ sentence-level results
• chunk_size=50 ⇒ paragraph-level results

LangChain usage

• Uses RecursiveCharacterTextSplitter
• Parameters:
  • chunk_size
  • chunk_overlap
• Code uses split_text() and observes the number/shape of produced chunks.

Video’s qualitative recommendation: This approach is described as the one people use most later, and it’s framed as the “best” among the four options.

---

3) Document-format-based splitting (language-aware / Markdown-aware)

Problem addressed

Plain-text splitting may fail for non-standard structure, like:

• Python code (keywords: class, def, braces, etc.)
• Markdown (headings, lists, sections)
• (Mentioned: could be extended to other formats like HTML/PDF-related formats)

Approach

• Extend the recursive character splitter idea, but use different separators appropriate to the document type/language.
• The “recursive” algorithm stays the same; only the separator strategy changes.

LangChain method (as described)

• Import a recursive splitter that supports language-specific separators (via from_language usage in code).
• Example workflow:
  • specify language = Python or Markdown
  • set chunk_size and chunk_overlap
  • call split_text() and inspect resulting chunks

Video’s Markdown demo

• Very small chunk sizes produced too many chunks.
• Increasing chunk size brought chunking closer to expected section boundaries.

---

4) Semantic-meaning-based splitting (SemanticChunker - experimental)

When it’s useful

When length/structure-based chunking fails because a single paragraph contains multiple unrelated topics.

Example from the video:

• First paragraph: agriculture farmers
• Then shifts to IPL
• Another section: terrorism
• Structure-based splitting might split into only 2 chunks, mixing unrelated topics inside one chunk.

How it works (conceptual)

Uses embeddings to detect topic shifts:

• Split text into sentences
• Embed sentences (example uses OpenAI embeddings)
• Compare similarity between consecutive sentence vectors
• When similarity drops abruptly (topic change), it inserts a split

Implementation details (experimental in LangChain)

• Uses SemanticChunker from langchain_text_splitters.experimental (as stated)
• Embedding model (OpenAI embeddings shown)
• Parameters mentioned:
  • threshold logic (example uses standard deviation)
  • threshold_amount (e.g., 1 vs 3 standard deviations)
  • threshold types listed: percentile, interquartile, standard deviation, gradient (video says more exist)

Observed behavior / trade-offs

• Lower threshold:
  • chunking can be inaccurate (e.g., a line about agriculture appears inside the IPL chunk)
• Higher threshold (e.g., 3 std dev):
  • may become too tolerant, producing fewer larger chunks (possibly a single chunk)
• Video conclusion:
  • promising, but experimental and needs tuning; not as reliable as recursive splitting.

---

Overall conclusion from the video

• Best practical default (among the four): RecursiveCharacterTextSplitter
• Semantic chunking can help with tricky semantic topic shifts, but it’s experimental and requires experimentation.

---

Main speakers / sources

• Speaker: Nitish (host of CampusX video)
• Primary source/library: LangChain (including LangChain community loaders and LangChain experimental SemanticChunker)