Video summary

I Built A Self-Improving AI Software Factory

Main summary

Key takeaways

Technology

Summary of Technological Concepts & Product Features

  • Software factory concept (for agentic coding): Infrastructure that runs coding-agent tasks through a defined, repeatable process, recording metrics and data each run so improvements can be measured over time.

  • Why it helps vs. “hype” agent swapping: Agentic coding is hard to optimize because changing models, prompts, harnesses, or skills makes outcomes inconsistent and difficult to attribute. A software factory enables controlled experimentation and repeatable measurement.

Architecture: Control Plane vs. Data Plane

  • Control plane: Manages work ingestion/queuing and orchestration.

  • Data plane (workers): Executes tasks via coding agents, sets up environments, performs work, and reports results back.

Key distinction:

  • A worker is the infrastructure/execution layer.
  • An agent harness is the agent definition/execution logic.

Scaling support:

  • Workers can be scaled across multiple VMs, Kubernetes, etc.

Security & Deployment Approach

  • Uses an SSH tunnel so the factory and agents can run on a private VM without public internet, while the developer accesses the UI locally.

UI / Operations & Analytics (Machinist UI)

  • Tracks completed/in-progress work.
  • Shows cycle metrics, such as:
    • Average task time (example: ~18 minutes)
    • Token usage (example given: millions of tokens)
  • Displays prompts / agent definitions.
  • Supports running multiple workers when needed.

Repeatable “Foreman” Workflow Prompt

  • Every task is processed by a foreman/coordinator that delegates steps to sub-agents in a consistent sequence.
  • Changing the main prompt/config affects every task, enabling systematic comparison.

Triggers to Bring Work Into the System

  • Repository monitoring health check: Scans the codebase with a cheaper model, identifies bugs/issues, and creates tickets.

  • GitHub label watcher trigger: Monitors for specific labels and pulls matching tasks into the factory.

Evals and Automated Testing Inside the Factory

  • Uses somewhat deterministic evals for automated checks/tests to verify:
    • correct workflow execution
    • correct labeling/assignment behavior
  • Supports ongoing evaluation as part of improving the system.

Structured Logging + Data-Driven Optimization (“Eval Loop”)

  • Agents emit logs / JSON metadata.
  • The factory maintains a historical record of runs.
  • Example evaluation task:
    • an agent reviews outputs of recent runs and suggests prompt improvements to reduce token usage.

Metric examples tracked:

  • times per issue
  • tokens used
  • commands run
  • agent weightings (as tracked data)

Cost / Token Reduction Idea

  • Switch certain steps from agent-only to more deterministic Python scripts:
    • Run scripts that monitor CI and delegate to agents only when needed.
  • Goal: cheaper, more predictable, more testable, and consistent structured output.

Operational Rollout Strategy

  • After validating improvements on the factory instance, apply the same workflow/config to the local development setup for day-to-day use.

Team Benefits vs. Solo Development

  • Factories provide shared infrastructure + shared workflows (similar to CI/CD):
    • teams improve prompts/config once
    • benefits propagate to everyone
  • Contrasts with teams using different models/harnesses/prompts, which leads to poor consistency and repeatability.

When Not to Use Factories

  • Avoid factories for highly interactive/iterative design work where you need fast back-and-forth and close code iteration—use local agents instead.

Best-Fit Use Cases

  • Batch/overnight work
  • Queuing large sets of tasks
  • Sequencing repeated workflows
  • Reliable execution followed by morning review

Tutorials / Guides / Setup Details Mentioned

  • Building and configuring a software factory, including:
    • Deploying on a VM (example: Ubuntu)
    • Using SSH tunneling for private execution + local access
    • Setting control plane + worker configuration
    • Defining prompt configurations (e.g., foreman + audit jobs)
    • Defining executors (e.g., Claude/Code* executors) and selecting which models are available
    • Configuring triggers (repo health check + GitHub label watcher)
    • Adding evals/tests and deterministic automated checks
    • Running improvement experiments by applying labels/tasks and measuring effects over days
  • An eval-driven workflow:
    • derive improvements (token efficiency, workflow correctness)
    • implement changes
    • re-measure

Main Speakers / Sources

  • Speaker: The video creator (solo engineer) discussing their own setup and results using “Machinist” (no specific name provided in subtitles).

  • Source/tool mentioned: Machinist (UI/CLI), along with coding-agent executors (e.g., Claude and “Code” executors), plus GitHub and CI*.

Original video