The Industry Is Catching Up: The Only Way to Generate AI Code That Works

Summary

Dave Farley argues that dependable AI-assisted programming depends less on generating code quickly and more on specifying and verifying what the software must do. He says the industry is converging on a key principle: the specification should be executable, not just a prompt or Markdown document.

Main Points

• AI speeds up code production, not the hardest parts of engineering. Developers still need to define intent precisely, verify the result, and work in small steps. AI can make these tasks harder: prompts are open to interpretation, agents may change more than requested, and rapid code generation can produce changes too large to assess meaningfully.
• Tests need an independent source of intent. If an agent writes tests after writing the code, those tests may only confirm what the code already does. Human-defined requirements are needed to judge whether the behavior is actually useful and correct.
• Markdown specs can drift from implementation. Farley discusses the rise of spec-driven development tools and approaches, including GitHub Spec Kit, Amazon’s Kiro, and proposals to maintain specifications as the primary artifact. He agrees with the goal but argues that prose-only specifications do not automatically detect when the implementation fails to meet them.
• Executable specifications link intent to verification. A behavioral example such as “Given a customer with a book in their basket, when they apply the discount code summer, then the basket total is reduced by 10%” describes the required outcome without prescribing the implementation. When the system does not meet that behavior, the test fails.
• Separate behavior from technical implementation. Farley describes a four-layer testing structure:
  1. Behavioral test cases written in the language of the problem domain.
  2. A domain-specific language (DSL) that makes scenarios consistent and reusable.
  3. Protocol drivers that interact with the system through its UI, API, or other interface.
  4. The system under test.

Keeping technical details in the lower layers helps behavioral specifications remain useful when the implementation, framework, or programming language changes.

• Treat the specification as the agent’s contract. The agent can choose how to implement the behavior, but should have no discretion over what behavior is required. Farley says acceptance tests should be protected from unapproved agent edits, since agents may otherwise delete failing tests instead of fixing the code.
• Small, incremental changes make errors cheaper to find. Farley rejects the idea that this approach requires writing a complete specification up front. Instead, each scenario is added incrementally, tested, implemented, reviewed, and committed.

Practical Workflow

1. Write one behavioral specification before the agent starts.
2. Run it and confirm it fails. The “red” result shows that the test can detect missing behavior.
3. Let the agent make it pass, then review and commit the small change.
4. Keep acceptance tests under human control. Treat changes to them as changes to the contract that require human approval.
5. Use mutation testing where appropriate. If altered code (“mutants”) still passes, the specifications may not cover enough behavior.

Farley notes that executable acceptance tests do not eliminate code review. Rather, they reduce the need to inspect every generated line by providing a stronger way to check whether the system meets its intended behavior. Poorly designed acceptance tests can still be brittle or provide false confidence, so he advocates separating behavioral scenarios from UI-specific details.

The video also mentions an ATDD skill for AI coding agents that Farley says guides users through building this layered testing approach. This is presented as a practical resource, not as an independent product review.

Main Speakers and Sources

• Dave Farley — presenter and principal commentator.
• Cited industry voices: Bob Martin, Adam Tornhill, Kent Beck, Emily Bache, Martin Fowler, Ian Cooper, and others.
• Organizations and tools discussed: OpenAI, Anthropic, GitHub Spec Kit, Amazon Kiro, Tesla, and Thoughtworks.