Verification is the new code review

Summary

As AI coding agents produce more code, verification is becoming more important than manually reviewing every line. Engineers are increasingly focused on planning changes and building reliable ways to confirm that the result works.

The verifier model

A verifier is a check that provides evidence an agent’s work is correct. It has three parts:

1. Surface: Where a user or another system encounters the change, such as a website, API, CLI, mobile app, or MCP server.
2. Answer key: An independently established definition of the correct result.
3. Comparison: A check of whether the observed result matches the answer key.

The answer key is crucial. If a coding agent writes both the implementation and its tests, those tests may simply reflect the code’s mistakes. Better answer keys include an existing system, an independent test suite, a saved snapshot, or a specification defining expected user behavior.

Examples of verification

• Anthropic’s messaging API rewrite: The team reportedly rewrote a Python API in Rust, despite not initially having Rust expertise. They used shadow traffic: the old and new systems received the same requests, and their responses were compared. Differences were investigated and corrected before traffic was gradually moved over. The reported benefit was a 40–50% reduction in CPU usage.
• Refactoring with an existing test suite: Bun’s rewrite from Zig to Rust used its TypeScript test suite as an independent answer key. The migration was completed with all tests passing and none skipped or removed.
• Exact-output checks: An agent-assisted decompilation of the game Crimsonland was verified by compiling the recreated source and comparing its machine code byte by byte with the original. For Ghostty, Mitchell Hashimoto used Kitty’s image-display behavior as a reference for compatibility.
• Visual comparison: Screenshots can serve as answer keys for interface reproduction. A pixel-difference comparison can show whether an agent’s version is converging on the reference design.
• New features: When there is no existing system to compare against, a specification can define expected actions and outcomes—for example, adding, filtering, and deleting tags in a note-taking app. Those behaviors can then become end-to-end tests.

End-to-end testing and agent workflows

The speaker recommends combining deterministic checks with agent-based end-to-end testing. In the E2E framework he discusses, one agent performs a workflow in a browser while a separate agent judges whether the expected result occurred. Separating the actor from the evaluator helps avoid having the same agent define and pass its own test.

He says he has removed large sets of unit and integration tests in some projects and replaced them with end-to-end tests tied to specified user behavior, reporting that this caught more bugs per pull request. He also warns that green unit tests can coexist with broken user-facing behavior, such as a button that does nothing.

Building and maintaining verification skills

The speaker describes three recurring tasks:

• Create a verification skill by identifying the product’s surfaces, possible answer keys, testing methods, and costs.
• Maintain it as the application and its platforms change. For example, tests that ran only in Chromium missed video playback problems in Safari. The verification routine was updated to include Playwright WebKit and a real Mac running Safari.
• Audit it to check that it actually catches defects. One suggested approach is mutation testing: intentionally introduce a realistic bug, then see whether the verifier detects it.

The overall developer workflow is a loop: the verifier finds a problem, the coding agent fixes it, and the verifier checks again. Bugs reported by users should also prompt improvements to the verification process so similar problems are caught earlier.

Takeaway

The central question before a major feature or refactor should be: “What is the verifier, and what is the answer key?” The speaker’s view is that engineers working with coding agents should devote substantial effort to designing, maintaining, and auditing verification systems—not just instructing agents to write better code or tests.

Reviews, guides, and tutorials

This is an explanatory guide to verification strategies, with examples involving shadow traffic, test suites, byte-level and pixel-level comparisons, end-to-end workflows, and mutation testing. The speaker also expresses a positive view of the E2E framework.

Main speakers and sources

Ray Amjad is the video’s narrator. He discusses an Anthropic podcast interview about the Rust API rewrite and references work by the Bun team, the Crimsonland decompilation project, Mitchell Hashimoto’s Ghostty work, and the E2E testing framework.