Video summary

Build an agentic software factory: deep dive

Main summary

Key takeaways

Technology

Summary of technological concepts & product/workflow details

  • Autonomous “software factories” / self-improving software

    • The core idea is a loop where the software continuously uses feedback and telemetry (e.g., error logs, uptime/performance metrics from tools like Sentry) to:
      • identify a bug
      • reproduce it
      • fix it
      • verify the fix
      • submit changes
      • ideally with minimal human intervention.

  • Agentic fix → verification → PR → ship cycle

    • The workflow described is essentially:
      1. Collect/monitor signals
      2. Delegate to agents
      3. Agents reproduce and fix
      4. Verify
      5. Open/update a pull request
      6. Validate via CI (and optionally human review)
      7. Merge/shipping
      8. Repeat
    • Agents can run locally or via hosted automation tools (e.g., Codex, Cloud Coder).
    • Once verified, agents create a pull request.

  • Human-in-the-loop vs agent-to-agent verification

    • The process includes multiple verification layers:
      1. A “standard review bot” comments on what the agent submitted.
      2. A separate PR-monitor/reviewer agent must review/resolve CI comments/failures, then either:
        • agree (comment/approve), or
        • disagree (explain why)
      3. Individual PR agents check:
        • CI passed
        • no merge conflicts exist
        • an approval policy approves merging
      4. Depending on configured rules, the system either:
        • requires human verification, or
        • proceeds using agent-based verification.

  • Policy configuration and scheduling

    • Factory behavior is driven by YAML configuration, defining:
      • Input sources (telemetry channels, GitHub issues, app feedback, internal feedback, etc.)
      • Policies controlling when an issue is safe enough for autonomous fixing
      • Hints and prompt-based guidance to steer behavior
    • Factories run using host scheduled tasks (hourly/daily/etc.).
    • The speaker mentions using Codex for scheduling (with similar capabilities in Cloud Coder).

  • Hosted “agent applications” (tools the agents can control like UI)

    • The speaker also maintains free/open-source hosted agent apps that are described as fully agentic (not small utilities, but capable of “anything a UI can do”).
    • Examples:
      • Figma-style editing/design creation
      • Google Slides-style slide editing
      • Calendar management
      • Mail and other productivity tasks
    • These apps can be used as visual agents, UI interfaces, or both.

  • Cost and model selection considerations

    • For high-volume PR generation, the speaker emphasizes cost efficiency:
      • running many PRs per day using Luna (described as very cheap; “runs on maximum settings” without hitting limits)
      • browser-based verification (screenshots/playback) can be token-expensive, so verification quality matters
    • For more complex oversight, larger models or subagents are referenced (e.g., Opus, Fable, Sonnet, etc.).

Key automations / “skills” mentioned

  1. Factory Collect

    • Collects and reviews input sources in a loop.
    • Decides when an issue is testable/reproducible and safe enough for autonomous fix + PR.

  2. PR Monitoring / Parent PR workflow

    • When an agent creates a PR, another automation checks it periodically.
    • The PR-creator/parent agent must ensure every CI comment/failure is addressed.

  3. Factory Watchdog Timer

    • Runs multiple times per day to detect stalled/incomplete work (e.g., review requests created but not sent, steps failing to progress).
    • Pushes/continues tasks until completion.

  4. Factory Inspection (weekly meta-analysis)

    • Searches for recurring patterns, such as:
      • repeated feedback types
      • repeated P90 degradations even after “fixes”
    • If patterns exist, delegates deeper analysis to a larger model to scan configurable windows (e.g., comparing last 30 days vs prior periods).
    • Uses triangulation across:
      • error/log data
      • customer feedback

  5. Discussion-thread “watchdog” / supervision

    • Monitors a linked discussion thread and sends instructions.
    • Intended to ensure the model follows the correct approach and can independently verify/guide work.

  6. Clarification automation

    • If reports lack detail, the system can request missing information automatically (e.g., URL, timestamp, logs).
    • Tracks whether clarifications were already answered before taking action.

Infrastructure / execution strategy

  • Run on a laptop vs cloud

    • Recommendation: run on a laptop, because real testing may require local environment access:
      • files/keys
      • browser sessions
      • real login flows
      • OTP prompts
    • Tradeoffs:
      • laptop may freeze
      • memory may be overloaded
    • Mitigation: use a second dedicated always-on laptop next to the main computer for continuous execution.

  • Remote task visibility

    • Mentions using Codex remote access to manage remote work trees and run code on the always-on machine.

  • Deployment strategy: beta vs production

    • Maintain separate beta and production environments:
      • Beta syncs with main; internal users test there.
      • Issues detected in beta are fixed automatically.
      • Production deploy happens daily to enable easier rollback.
    • This reduces risk compared with very frequent automated deployments (e.g., every 10 minutes).

Practical “advice” / how to start small

  • Start small and build up

    • Begin with a minimal factory:
      • take an input source
      • pass a finite set of tasks to agents
      • require high-confidence verified changes, with human review priority
    • Configuration is YAML-driven; the agent can iteratively update its YAML config based on feedback.

  • Repository structure

    • Keep everything in one repository (a “single isolated mono repository”) to simplify factory behavior.

  • Input data quality is decisive

    • Invest in:
      • robust telemetry and customer feedback channels
      • clear bug visibility tied to original source code (e.g., via source maps)
      • end-to-end tests on beta builds
      • fast conversion of failures into logged issues for immediate resolution

  • Skills distribution

    • Use a skills repository, plus a CLI installer to distribute/customize skills and configure automation.
    • Mentions “step zero” as running skills locally (e.g., “factory build”, trial runs, and configuration iteration).

Named product/platform references

  • Codex, Cloud Coder
  • Sentry
  • GitHub issues and pull requests
  • Vercel (example of automated deployment error fixing)
  • Figma-like design tools, Google Slides-like editing
  • Vercel deployment, beta/production environments
  • Agent Native Framework (and agentnative.com for apps)

Main speakers / sources

  • Primary speaker: the video author/speaker describing their own open-source agent project and a “skills repository.”
  • No additional distinct speakers are clearly identified in the subtitles.

Original video