Video summary

Lauren Tan workshop XAi Grokbot

Main summary

Key takeaways

Technology

Tech concepts & product features discussed

  • Trust in code-writing agents (core theme): Lauren argues that the limiting factor when using coding agents is trust. Without trust, engineers stay “in the loop” (micromanaging outputs), can’t parallelize many agents, and can’t safely scale automation.

  • Verification as the key skill for agent trust: She emphasizes that agents become trustworthy when they can verify behavior by actually running the system, not just editing code. Verification examples include:

    • Running code/tests
    • Taking/using CPU traces
    • Using heap snapshots
    • Controlling apps via simulators (e.g., iOS simulator)
    • Using tool protocols (e.g., Chrome DevTools Protocol)

  • Cursor’s “agents window” control & performance tooling (Control Glass / Glass):

    • Cursor internal code name: “glass” for the Agents Window UI.
    • A major issue: the agent could control the application, but didn’t understand where UI features are, so it “flailed” navigating.
    • Solution: Feature Map
      • A special “feature map” file teaches the agent how to navigate to UI features and interact with DOM elements / CDP attributes.
      • Improves response to low-quality user reports (often screenshots + “?”), because the agent can map what the user is pointing at.

  • Pstack (Pac? / “potato stack”) plugin

    • Presented as an incrementally built set of skills for improving agent reliability.
    • Purpose is largely to reduce hallucination by forcing agents to:
      • search code,
      • run tools,
      • use sub-agents,
      • and stop guessing.
    • Uses the “skill as markdown with embedded instructions/tooling context” approach.

  • Creating & maintaining verification via evals

    • Lauren likens evals to unit tests for agents/skills.
    • Cursor ships an “eval playbook” (under “potato mode”) that:
      • spawns many sub-agents,
      • uses a coordinator + rubric,
      • runs tests in ways meant to avoid agents “detecting” evaluation mode,
      • evaluates skill performance across a matrix of supported models.
    • She also describes “hill climbing” an eval using loops (e.g., repeatedly running until you reach target scores).

  • Scaling from local to cloud agents

    • Practical guidance:
      • Start verification locally so you can observe tool calls and application interactions.
      • Only scale up after trust is established.
    • Cloud agents example: “Benny”
      • Takes bug reports, runs in a cloud desktop environment, reproduces issues, and provides evidence (sometimes confirming “already fixed on main”).
      • Benefits: faster diagnosis and less manual investigation for engineers.

  • Codebase constraints to enable safe agent scaling (Dune / Grockbot architecture)

    • She argues that beyond verification, hard constraints in the codebase are critical.
    • In Grockbot, CI is made “strict/annoying” to prevent known agent failure modes, including:
      • banning specific patterns like React useEffect
      • banning code comments (agents add irrelevant or misleading comments)
      • enforcing strict directory/process boundaries for Electron apps:
        • separation between electron main and electron renderer
        • CI checks dependency graphs so agents don’t accidentally import/run heavy code in the renderer thread and cause performance regressions/jank.
    • Architectural analogy: strong constraints turn agent coding into something like “chef design the kitchen” vs manually cooking everything.

  • Refactoring/rewrite argument

    • She claims “vibe-coded” greenfield prototypes lack guardrails and become messy as agent-driven automation expands.
    • She makes a case for controlled refactoring/rewriting when you can introduce constraints and CI safeguards so agents can operate safely.

  • Grockbot product mention

    • Grockbot introduced as a new application (launched “yesterday/today” in the talk).
    • Described as allowing orchestration of individual agents with identities, in an accessible interface.
    • She compares it to a “Kusher moment” / iMessage-like UX for non-developers (PMs/designers), enabling broader adoption and faster feature shipping.

  • Product metrics / operational details

    • She reports high agent-driven PR velocity at Cursor (hundreds to ~1k PRs landed across months), attributing it to improved verification + constraints.
    • PR sizes: no hard cap; commonly ~50–1000 lines, varies by task.
    • Encourages splitting work into multiple PRs for atomic history and easier reverts.

Key takeaways / tutorial-like guidance

  1. Build trust via verification: require agents to run the app/test via real tooling (traces, snapshots, simulators, CDP control).
  2. Add navigational grounding for UI automation: Feature Map so agents can interpret user reports and locate UI elements.
  3. Use evals like unit tests: create rubrics, evaluate across model matrices, iterate until scores hit targets.
  4. Scale only after local trust: start local, observe behavior, then move to cloud agents (e.g., auto-repro via “Benny”).
  5. Enforce hard constraints in CI/architecture so agents can’t “copy bad patterns” (e.g., renderer/main separation for Electron, banned patterns like useEffect).
  6. Refactor “vibe-coded” greenfield systems to add guardrails that agents can follow reliably.

Main speakers / sources

  • Lauren Laurent Tan (Twitter identity: “Potato”; also referred to as “Potato”/“potato mode”) — main speaker, with background at:
    • Meta (React compiler / core team)
    • Netflix (tech lead → engineering manager)
    • Cursor (~5 months)
  • Colin — moderator/host who asks questions and prompts discussion.

Original video