Every Level of a Claude Second Brain Explained

Overview

The speaker explains how to build an “AI second brain” as an incremental set of data-routing and retrieval architectures—from simple folder-based keyword lookup to autonomous always-on memory systems.

The core goal is not only storing information, but ensuring agents can reliably retrieve the right information, reducing hallucinations and wasted tokens.

---

Key concept: “Moat is your data,” but retrieval/organization matters

• The value is getting knowledge out of your head into a system.
• The bottleneck is organizing that data so AI models can recall it correctly and efficiently—not searching everything or letting the model hallucinate.
• This is framed as:
  • relationship mapping
  • routing-based retrieval
  • instead of brute-force scanning

---

The 5 Levels (what improves, and what question each one answers)

Level 1 — Exact lookup + routing (files/folders as the system)

Question: Can you find the info by an exact word/name?

What it uses

• A project structure where claude.md (or agents.md for Codex) acts like a router/system prompt.
• Markdown folders such as:
  • context/
  • projects/
  • decisions/

How routing works

• Routing rules tell the agent where to look.
  • If the user asks about “me,” check a specific folder.
  • If asking about “quarter priorities,” route to the right place.

Downsides

• As it grows, routing can become messy.
• It tends to behave more like keyword/exact-match retrieval.

Example folder contents

• about me (expandable)
• stack and conversations
• decisions as an append-only decision log (dates + changes)
• projects optionally organized by date/client/project

Review/guide theme

• If the AI is missing info it “should know,” you likely need better routing and folder architecture, not a new model.

---

Level 2 — Topic consolidation + wiki + auto memory file

Question: Can you pull everything on a topic together?

What it adds

• An LLM Wiki layer (similar in spirit to a “Karpathy LLM Wiki” approach).
• Transcripts and research live in wiki structures that cross-link “main concepts.”

Auto-created indexes

• Claude code ingests content and generates related pages such as:
  • sources
  • platforms
  • context techniques
  • etc.

How it builds on Level 1

• Structure is still similar, but claude.md routes additionally to:
  • references and memory.md

Auto memory feature (Claude Code specific)

• Claude Code can enable auto memory, so the AI maintains memory.md automatically.

Tool-agnostic note

• Claude Code uses claude.md + memory.md.
• For Codex, rename/copy claude.md → agents.md, and configure routing so Codex uses the same memory.md.

Important nuance: wiki links ≠ knowledge graph

• Wiki relationships are more like:
  • backlinks/see-also trails
• It’s not the same as a true semantic/typed relationship graph.

---

Level 3 — Semantic search + vector databases (meaning-based retrieval)

Question: If I search using different words, can you find it anyway? Can you retrieve by meaning?

What it uses

• Semantic search rather than exact keyword matching.
• Demonstrated via a vector-search “smart lookup” UI:
  • Keyword search (“feedback”) returns exact matches
  • Semantic search (“feedback”) returns conceptually similar chunks (e.g., “evaluation” / test results)

Vector DB mechanics (as explained)

• Chunk documents (e.g., transcripts)
• Embed chunks and store vectors
• Retrieval returns the most similar chunks, not the whole document

Critical limitation (full-context problem)

• Chunk-based retrieval can miss important details if it only surfaces the “top relevant chunks.”
• Example failure mode:
  • “highest sales week” may be wrong if only one chunk is retrieved rather than all relevant context.

Design guidance: mixed architectures

• You don’t have to use semantic search everywhere.
• Keep some folders as markdown/context files for full-document accuracy.
• Use vector retrieval where you have high-volume material.

Practical recommendation

• Use semantic/vector retrieval for:
  • precise snippets from large corpora (e.g., “rule 17 out of 1000”)
• Use markdown end-to-end reads when:
  • full context is necessary for accuracy

---

Level 4 — Knowledge graphs / relationship graphs (entities + relations)

Question: Can you model explicit entities and how they relate over time and across concepts?

What it introduces

• Knowledge graphs:
  • Entities (e.g., Jordan = person, Acme = company)
  • Relationships (e.g., “Jordan works at Acme,” “Acme endorsed by Postpilot,” etc.)

Complexity/cost

• Can be expensive/complex.
• The speaker doesn’t use it daily, but suggests it can make sense when:
  • data has lots of structured relationships

Data-ingestion strategy (“Grill Me” skill)

• A skill that “interviews” the speaker relentlessly to produce brainstorm files for a topic.
• Goal: generate enough structured understanding to seed a knowledge graph.

Tradeoff vs wiki

• Wikis offer navigable trails via links.
• Knowledge graphs model relationships more explicitly.
• Building graph edges may require reading/embedding entire relevant files/entities.

Reference implementations mentioned

• Mentions the Graphir / Logseq-style ecosystem as possible approaches.

---

Level 5 — Always-on autonomous “Brain OS” (continuous syncing)

Question: Can the second brain run itself and keep updating without manual actions?

Inspiration

• Inspired by “Gbrain” (Garry Tan / Y Combinator context), paired with an agents/tool ecosystem (“G stack” mentioned).

Always-on behavior

• Continuously syncing, refreshing memories, and adding updates.

Concern: context overload

• Fear that “too much context” can harm performance.
• The speaker prefers controlled ingestion:
  • run skills to fetch weekly meeting transcripts
  • ingest only what should be retained long-term

Core framing: “4 C’s” and data types

• The second brain focuses mainly on:
  • Context: evergreen operational decisions (e.g., quarter OTAs)
  • Connections: less evergreen real-time info (e.g., Slack threads/emails)
• Avoid continuously ingesting high-churn “noise.”
• Instead, retrieve noise on demand from live systems.

Retrieval order described (routing across sources)

1. Check OTA/project file (context decisions)
2. If missing, search wiki/meeting transcripts
3. If still missing, pull from ClickUp / live threads for current conversation context

---

Additional product/practice guidance (beyond the levels)

• Your system doesn’t need to be one level. Different folders can use different architectures (markdown routing vs wiki vs vectors vs graphs).

• Start based on your pain:

  • Missing exact-known files → Level 1
  • Too many notes / can’t find what’s where → Level 2
  • Retrieval fails due to vocabulary mismatch → Level 3
  • You need explicit relational chains/entities → Level 4
  • You want continuous autonomous updates across agents → Level 5

---

Main speakers / sources

• Main speaker: The video author/instructor, presenting their “Hercule/Herc 2” second brain projects.
• Referenced sources/products/people:
  • Garry Tan (Y Combinator) — Gbrain
  • Karpathy — “Karpathy LLM Wiki” reference
  • Matt Pocock — “Grill Me” skill
  • Tools/platforms mentioned:
    • Claude Code
    • Codex
    • Hermes Agent
    • Obsidian
    • vector databases (e.g., via Pinecone/Supabase mention)
    • Anthropic/Claude (implied processing)
    • ClickUp