How To Implement Short Term Memory Using LangGraph

Video goal

Show how to implement short-term memory (STM) in LangGraph for agentic LLM applications, then address:

1. Persistence (so memory survives restarts)
2. Context overflow (LLM context window limits) using:
   • edge trimming
   • deletion
   • summarization

---

1) Short-term memory implementation in LangGraph (concept + code pattern)

Core idea

• LLMs are stateless: each llm.invoke behaves like a new conversation unless you supply prior context.
• Implement STM by maintaining a conversation buffer:
  • Send the prior conversation history alongside the latest user message.

LangGraph mechanism used

• Checkpointer: stores the graph state at each “superstep”
• Thread ID / thread: isolates memory per conversation

Practical behavior demonstrated

• With the same thread ID, asking “What is my name?” works because earlier messages were checkpointed and reused.
• Changing the thread ID starts a “new conversation,” so the model again says it doesn’t know the name.

Main features shown

• Use InMemorySaver checkpointer (RAM)
• Provide configuration with thread_id
• Inspect graph state to see stored messages per thread

---

2) Persistence: fixing RAM volatility (Postgres via PostgresSaver + Docker)

Problem

• In-memory checkpoints vanish on restart:
  • When the program restarts, all stored checkpoints in RAM are lost.

Production-grade approach

• Use Postgres as the backend for checkpoints.

Tutorial steps included

• Install and run Docker Desktop
• Create a docker-compose.yml for Postgres
• Install Python dependencies (LangGraph + Postgres checkpoint integration + model libs)
• Use PostgresSaver as the checkpointer and provide a DB connection URL
• Use a context manager so the saver/connection is used properly

Verification shown

• Start a conversation, restart the program, and then fetch prior state without reinvoking:
  • The previous messages still exist for the same thread_id.

Key concept

• Replace InMemorySaver with PostgresSaver to persist STM across sessions.

---

3) Context overflow problem (context window overrun)

Recap

• Every LLM has a context window (max tokens it can process).
• If STM keeps appending full history, tokens can exceed the limit, leading to unreliable outputs and possible hallucinations.

Techniques to manage token growth

A) Trimming (keep last messages within a max token budget)

• Set a maximum token limit
• Before calling the LLM, run trim_messages
• If history exceeds the limit, keep only the most recent messages whose combined tokens fit

Important detail

• Trimming does not delete messages from state—it only prevents old messages from being sent to the LLM.

Demonstrated limitation

• Because omitted context is not available to the model, the conversation may lose important information and “break” continuity.

B) Summarization (summarize old messages instead of dropping them)

• Motivation: trimming may discard useful context entirely.

Strategy

• When history grows:
  • Send the most recent N messages
  • Plus a summary of older messages
• The older content is summarized by another LLM step
• The summary is prepended as a system message for future calls

Requires deletion logic

• Avoid duplicating both old messages and summaries:
  • After summarizing, delete older messages from state and keep only:
    • the summary
    • the most recent messages

---

4) Deletion + Summarization workflows (code walkthrough logic)

Deletion node

• Uses LangChain removeMessage to permanently delete items from graph state.
• Example behavior:
  • If message count exceeds a threshold (e.g., >10), delete the oldest messages and keep only the newest portion.

Summarization node (summarize + delete together)

• Custom state includes:
  • regular messages
  • a summary key

Workflow

1. Chat node

   • If summary is empty: send only messages
   • If summary exists: send summary + recent messages

2. Summary node

   • Trigger only when len(messages) > 6 (conditional logic)
   • If a summary already exists: merge/extend it with a summary of new overflow messages
   • Delete overflow messages, leaving only:
     • the last two recent messages
     • the updated summary

---

Key takeaways

• STM in LangGraph: implemented via Checkpointer + thread ID.
• Persistence: use PostgresSaver instead of in-memory checkpointing to survive restarts.
• Context window management:
  • Trimming prevents overflow by sending only recent messages, but may lose context.
  • Summarization + deletion preserves meaning via a rolling summary of older conversation.

---

Main speakers / sources

• Speaker: Nitesh (host), presenting a LangGraph tutorial and code walkthrough.
• Sources used in implementation:
  • LangGraph (checkpointers: InMemorySaver, PostgresSaver)
  • LangChain utilities (e.g., trimMessages, removeMessage)
  • Postgres (via Docker Compose) for persistence