Fixing the PR Bottleneck — Matt Pocock, AIHero

Business-focused summary: Fixing the PR bottleneck with AI-enabled “software factories”

The talk frames PR throughput as an operational bottleneck that worsens when AI agents generate more changes than humans can review. The solution is an engineering process “factory” with automation + quality brakes—so human review becomes faster and focuses only on high-risk/irreversible changes.

---

Core strategy / operating model: the “software factory” with brakes

• Software factory (process): offload parts of PR initiation and change execution to agents (e.g., classifiers or automated fix flows triggered by related systems like database monitoring).
• Deterministic automation increases throughput, but must be controlled to avoid generating unreviewable output (“garbage cannon”).

Three “brakes” / layers of quality

1. Automated checks (deterministic CI-style gates) Includes linting, tests, type checking, and code quality.

2. Automated review (agent-based analysis) A review agent inspects code structure and issues not caught by tests.

3. Human verification (final gate) Humans review only the reduced set of high-value changes.

---

Frameworks / playbooks explicitly described

“Cake” of verification (stacked gates)

Automated checks → automated review → human verification

Goal: make human verification faster by improving the earlier layers.

Red-Green-Refactor (two-step quality process)

• First context window: “make it work” (red/green)
• Second context window: “make it good” (refactor + coding standards)

PR risk classification using AWS “one-way door” vs “two-way door”

• Two-way door: safe to merge and roll back if needed → less review burden.
• One-way door: irreversible consequences (e.g., sending emails to 60,000 users, expensive migrations, data loss) → maximum scrutiny.

“Retro” (retrospective learning loop)

• Collect recent PRs/reviews and generate improvements to automation and standards for the next cycle.
• Principle: “never write the same comment twice.”

---

Concrete operational recommendations (actionable)

1) Improve quality earlier so humans review less

Principle: higher release quality reduces human review time because fewer interventions are needed. Add more automated checks because they are “cheap” relative to human time.

2) Treat automated checks as fallible (“lie detectors”)

CI green status doesn’t guarantee readiness; automated gates can create false confidence.

Examples of “lies”:

• Tautological tests Tests that mirror the implementation rather than validating behavior. Example: if the character limit is 280, the agent writes a test like “expect character limit for post X to be 280” (implementation-coupled, hard to refactor).

• Structure-sensitive tests Tests that inspect source/module ordering rather than rendering/behavior. Example: parsing source files to ensure the video section appears after the content plan can break when source organization changes.

• Tests that cannot fail (via stubbing/mocking) Stubbing DOM/audio APIs may avoid exercising complex error modes/crashes, allowing production issues to slip through.

Mitigation: raise the quality of inspections by using better code structure and better review tooling.

3) Use “code-based design” to make bad tests harder

• Deep modules: hide complex behavior behind a simple interface so agents/testers interact with interfaces rather than internals.
• Design goals tied to the talk’s concepts:
  • Locality: change one part without chain reactions.
  • Leverage: callers get lots of value from a simple function.
  • Seam: boundaries enable change without rewriting everything.
• Effect on agents: agents should operate through interfaces (“don’t dig into implementation to test weird details”).

4) Don’t overload the implementation agent—enforce standards in a separate review step

Key advice: don’t enforce coding standards inside the main implementation agent; it causes context-window overload and reduced performance.

Instead:

• Create a code review agent that receives a diff, then checks against coding standards files.
• The review agent may do research for context but doesn’t do debugging/implementation.

Implementation detail:

• Store standards in coding standards.md rather than global docs like agents.md, so only the review agent processes them.

5) Build internal automation rather than relying on third-party generic bots

Outsourcing automated testing is hard to get right universally:

• Too general → false positives
• Too specific → limited to certain languages/contexts

Recommendation: develop internal coding standards and an automated reviewer over time.

6) Make the PR itself human-friendly

Develop “PR skill” over time:

• Prioritize which checks matter (not all inspections are mandatory).
• Add a short “merger risk” summary at the end of PRs (e.g., “two-way door”, “radius local”, etc.).
• Improve clarity of what’s changing:
  • Use pseudocode and standardized diagrams (Mermaid/UML).
  • Prefer “show me” style (diagrams/images) over long text.

7) Close the loop with “retro”

Retro analyzes:

• automated checks + coding standards improvements
• navigability (how easily agents found information; whether hints in agents.md help)
• tool effectiveness and token usage
• redundancy (overloaded control files / excessive skills)

Outcome: cumulative improvement so future reviews require less work.

---

KPIs / metrics / targets mentioned

No explicit numeric business KPIs (e.g., revenue, CAC, LTV) were provided.

Implied operational metrics include:

• PR review throughput / speed (human verification becomes faster)
• Number/size of PRs humans must touch (reduced via automation quality)
• Error/defect leakage from automated checks (preventing “green CI lies”)
• Token efficiency (analyzed via retro)

No specific time targets were stated (e.g., “reduce review time by 30% in 6 weeks”).

---

Case examples / evidence used in the talk

• Examples of bad test patterns from agents:
  • tautological “character limit = 280” test
  • source-order structural test for interface modules
  • stubbing audio API so crash/error modes aren’t exercised
• Operational example of risk:
  • “simple change sending email to 60,000 people” as a one-way-door scenario
• Tool/service examples mentioned:
  • Cursor Bug Bot
  • Code Rabbit
  • (Speaker’s view: internal build is better)

---

Presenters / sources

• Matt Pocock (AIHero)
• Mentioned external sources/ideas:
  • John Osterhout (referenced for deep module / software design philosophy)
  • Dex Hadley (“show me” skill from a “Human layer skills repository” )