Video summary

Building Context Graphs for AI Agents, Will Lyon, Neo4j

Main summary

Key takeaways

Technology

Context graphs (the missing “why”)

  • The talk frames context graphs as a way to capture the missing “why” behind AI decisions. For example, if an agent recommends credit approval, it should also explain why—including the exact data and policies used.

  • A context graph is described as a knowledge graph containing the organization’s decision-relevant information, not just the final output.

Operations example: auditability “beyond what happened”

  • Audit logs can show what happened (e.g., a rejected transaction).
  • A context graph stores the decision components (e.g., the relevant policy/logic) that caused the outcome.
  • It also combines information from many systems into one queryable graph.

Key idea: store the causal/decision ingredients that justify outcomes—not only the outcome itself.

Financial services data model example

Model components

  • Entities
    • People / organizations
    • Accounts
  • Events
    • Transactions
  • Context / why
    • Decisions and policies (modeled explicitly)

Precedent links (causal/history traversal)

  • Decisions can connect to other decisions as precedents, enabling traversal through causal or historical reasoning chains.

Live demo app: credit-limit scenario

  • Demo app hosted online: sell.app
  • An AI “context graph agent” takes requests like: “Jessica Norris requests a $25,000 credit limit.”

Agent capabilities

  • Includes a system prompt (e.g., “helpful agent that knows a lot about financial services”).
  • Uses tools to interact with the context graph.

What the UI shows

  • The UI renders tool calls and retrieved graph components.
  • Analysts can:
    • Traverse the graph (people → decisions → policies)
    • Inspect causal chains that lead to the recommendation

Recording new precedent

  • The agent produces an outcome such as conditional approval.
  • It can then record the decision back into the graph, adding new precedent nodes for future reasoning.

Retrieval approach: hybrid search (graph + vector)

The agent’s workflow:

  1. Search for the customer in the graph.
  2. Traverse to gather surrounding context.
  3. Use hybrid search, combining:
    • Vector similarity (semantic relevance)
    • Graph traversal/structure relevance

Neo4j tooling mentioned

  • Graph Data Science
  • Example algorithm: FastRP for graph embeddings
    • Embeddings capture not only text semantics, but also structural patterns (e.g., relationships between accounts and transactions that reflect fraud patterns).

Goal

  • Find the most relevant precedents and policies by combining embedding-based retrieval with graph structure.

Building agents: capturing reasoning traces

A core challenge is capturing reasoning traces (e.g., tool-call history and decision steps) for:

  • auditing
  • future context reuse

neo4j-agent-memory

  • Neo4j offers neo4j-agent-memory (open source Python package).
  • Adds graph-based agent memory to agents.
  • Works via integrations with multiple agent frameworks.

Relationship to context graphs

  • Memory helps build the context graph and query it for decision-making.

Three memory abstractions used for context graphs

  1. Short-term memory

    • Conversation/session state (“working memory”)
    • When a message is saved, background processing moves relevant info toward long-term storage.

  2. Long-term memory

    • Extracted structured knowledge added to the graph
    • Uses entity extraction to convert unstructured conversation into graph entities and relationships.

  3. Reasoning memory

    • Stores reasoning/decision traces
    • Keeps tool call traces and reasoning steps in the graph.

Entity extraction pipeline (cost/performance emphasis)

The package emphasizes not relying solely on LLM-based entity extraction, due to being slow and expensive.

Pipeline components

  • spaCy for named entity recognition
  • Glyner 2 (local small model) for extraction and relationship extraction
  • LLM fallback only when needed

Data model emphasis

  • Extracted memory should be domain-relevant
  • Default extraction schema: POLE + O
    • People, Organization, Location, Event + Object
  • Users can apply their own domain model.

Connected graph architecture

  • Short-term conversations, extracted entities, and reasoning traces are stored in one connected graph.
  • This enables querying across:
    • extracted context from multiple systems
    • conversation history
    • reasoning traces / tool results that produced decisions

Second demo: “Lenny’s memory” podcast graph

  • Demo app uses Lenny’s podcast transcripts.
  • An agent-memory-based extraction process constructs a context graph.

Example queries

  • Find locations mentioned in specific episodes (e.g., Brian Chesky episode).
  • Geocode + enrich locations using external sources (e.g., Wikipedia).

Takeaway

  • Context graphs can include enrichment for non-text data, such as geospatial information.

Operating at scale + observability

  • Reasoning traces are written back into Neo4j cloud (example: Neo4j Aura).
  • In Neo4j Browser / Query Workbench, users can inspect:
    • extracted entities
    • recorded reasoning traces as sequences of tool calls
    • metadata such as token counts and tool responses

Integrations and other supported features

Neo4j agent memory integrates with many agent frameworks and cloud environments, including:

  • Google ADK
  • AWS (e.g., references to Bedrock/Vertex)
  • Microsoft agent framework

Additional mentioned capabilities:

  • MCP
  • OpenTelemetry support
  • OPIC (referenced in subtitles)

Agent swarm concept (shared memory layer)

  • A resource mentioned: a multi-agent (swarm) version of the financial services demo.
  • Multiple agents with different personas (e.g., compliance, anti-money laundering, customer service) share the same Neo4j agent memory/context graph layer.
  • This enables shared context retrieval and contribution.

Community/event callout

  • Promotes Nodes AI (Neo4j online conference).
  • Encourages registration via QR code.

Main speaker(s) / sources

  • Will Lyon — Product Manager at Neo4j, AI Innovation team (main speaker)
  • Neo4j — product/context (Neo4j Graph Data Science, Neo4j Agent Memory)
  • sell.app — context graph demo source (referenced)
  • Emil — mentioned as joining the end panel (in audience)
  • Foundation Capital — referenced source discussing the “trillion-dollar opportunity” for context graphs

Original video