Video summary

Deploying AI on the Buyside: A 20-Year Engineer's Playbook

Main summary

Key takeaways

Finance

Core themes (buyside + AI deployment)

  • “Single-player” → “multi-player” AI deployment: moving from individual experimentation to a firm-wide system depends on where data lives and how it’s structured, captured, and shared (not just model quality).
  • “Digital company brain” / talk-to-your-data: firms need to operationalize institutional knowledge (structured + unstructured) so analysts can query it across teams.
  • Data + workflow packaging beats “magic wand” ingestion: repeatedly emphasized that you must know what output you want and how to validate it; dumping everything into an LLM usually fails due to:
    • context limits
    • quality degradation
    • poor/insufficient instructions

Step-by-step / methodology frameworks mentioned

Job-to-be-done approach for enterprise AI rollout

  1. Identify the firm’s current workflows and what analysts produce/share.
  2. Discover data sources and where notes/data are stored.
  3. Choose one or two workflows to systematize/operationalize first (incremental progress).
  4. Create the “wiring”: ensure data is accessible to tools/agents and usable in the intended processes.
  5. Drive process change adoption across the team (capture new/standardized data if needed).

Task segmentation to handle context window limits

  • Preprocess and split large tasks into smaller subtasks.
  • Use a pipeline where each stage fits within manageable context and improves output quality.

RAG/vector search pattern for large unstructured documents

  • Extract text from documents (e.g., filings/PDFs).
  • Store in a vector database.
  • Search relevant passages, then have the model synthesize/distill.

Skill creation via “hill climbing” (iterative red-pen loop)

  • Build a workflow/skill that produces output.
  • Inspect outputs for what’s good/bad.
  • Iterate on instructions/skills using feedback loops.
  • Add evaluation mechanisms where possible; for subjective text outputs, rely heavily on human review feedback.

Key finance-relevant implementation challenges

  • Excel / large financial models are hard to “just connect”

    • Example: a model connected SharePoint to ChatGPT expecting it to interpret financial models, but it wasn’t “nowhere close.”
    • Common issues cited:
      • token/context constraints
      • quality degradation as more text is pushed into context
    • Recommendation: don’t ingest the whole workbook at once; instead segment tasks and preprocess/pipe outputs.

  • Risk of misaligned “accounting vs investing” semantics

    • Example: LLMs may treat goodwill impairment as a “scary risk” due to accounting terminology.
    • Fix: explicitly disambiguate and constrain output so goodwill impairment isn’t treated as a fundamental investment risk signal if that’s not the analyst’s intent.

  • Non-determinism caution for quantitative/production use

    • LLM outputs are non-deterministic; for calculations or determinism-required steps, use code / traditional computation rather than relying on LLM reasoning.

  • Production “last mile” accuracy/consistency/token economics

    • Prototyping with open tools is fast, but:
      • accuracy and consistency can degrade in production
      • token economics become problematic when scaling from demo → deployment

Explicit recommendations / best practices (as stated)

  • Preprocess and retrieve instead of brute-force prompting

    • For large unstructured inputs (thousand-page PDFs), avoid “pointing” an LLM directly at the PDF.
    • Prefer: extraction + vector database + search + targeted distillation.

  • Use “skills” as durable, reusable workflow artifacts

    • Convert successful experiments into reusable skills.
    • Share skills across the firm to reduce duplicated work and improve adoption.

  • Use LLMs to help write down intent/process

    • If analysts struggle to document their decision process, record a voice memo, transcribe it, and prompt the model to ask exhaustive clarifying questions until intent is explicit.

  • Human-in-the-loop evaluation for narrative outputs

    • For written reports: review with the consumer (“step through this memo”) and capture feedback to improve the upstream skill.

  • Limit custom engineering; prefer stable “harnesses”

    • Build on main copilots/harness tools (avoid bespoke tooling everywhere).
    • Connect via standardized integrations (e.g., MCP-style data connectors).

Tooling and infrastructure concepts mentioned

  • Vector database / RAG: retrieval from large document sets.
  • Skill libraries / skills
    • Includes “skills creator” concepts; caution against over-reliance without output review.
  • Harnesses / orchestration tools
    • Codex and Coda Work repeatedly referenced as central operating environments.
    • OpenClaw and Hermes mentioned as alternatives/competing approaches (OpenClaw described as having frictions like cost and features).
    • Perplexity computer: cited as more consistent; described as an “abstracted open claw” with access to models and subagent execution.
  • MCP servers: integration pattern for connecting skills to external APIs/data sources.
  • Local/server + permissions model
    • Codex permissions discussed as local to a machine, implying an always-on local setup may be needed for delegated tasks.

Key numbers / quantitative details mentioned

  • 1.2 million tokens: example scale of an attempted SharePoint → ChatGPT ingestion of a large financial model (implied too large / context-quality limited).
  • “Thousand-page PDFs”: example unstructured input size where preprocessing + RAG becomes necessary.
  • ~100x cost improvement: example cited (Shopify) after optimizing a workflow previously run on a frontier model.
  • “80% book decline”: Tim Ferriss tweet reference—books declined ~80% over 5 years, attributed to AI summarization effects.
  • Context-window note: “million token context windows” exist, but quality degrades when too much is included.

Assets / tickers / instruments mentioned

  • No specific market tickers, ETFs, bonds, commodities, or sectors were named.
  • Mentions were operational/conceptual (SEC filings/transcripts, Excel models, earnings preview reports, financial reports) without concrete securities.

Disclosures / disclaimers

  • No explicit “not financial advice” or regulatory disclaimer appears in the provided subtitles.

Presenters / sources mentioned

  • Matt Stockton (primary guest)
  • Brett (host/another speaker; last name not provided in subtitles)
  • Andrej Karpathy (referenced for AI insights)
  • Tim Ferriss (referenced for books anecdote)
  • Anthropic, OpenAI (organizations referenced for tutorials/docs)

Original video