Quality Signal, Not Vanity Metrics: An Interview with Bas Dijkstra, CypressConf 2026 Keynote Speaker

Summary

In this interview, test-automation consultant Bas Dijkstra argues that teams should judge their testing by the quality of the information and confidence it provides, not by vanity metrics such as test counts or code coverage.

Key points

• Mutation testing measures whether tests can detect changed behavior. A tool makes small changes to product code—such as inverting a condition or removing a function call—and reruns the test suite. If no test catches a change, that can reveal a gap in the tests. Mutation coverage—the share of mutants detected—is useful, though Dijkstra cautions that no single metric is a definitive measure of quality.
• Code coverage alone is a weak signal. A high line-coverage percentage shows that code ran, not that tests would detect an incorrect result. When a team reports high coverage, Dijkstra’s first question is essentially: What do you know now, and is that enough to make a decision?
• Testing should support decisions about product risk. Rather than report only how many tests ran or failed, testers should explain what they observed, how it could affect users or delivery, and what risks decision-makers are accepting.
• Test reliability should be made visible, not hidden. Re-running failures until they pass can mask useful evidence: a failure might point to a product defect, a test issue, or problems with test data, the environment, or infrastructure. Dijkstra recommends tracking test-suite volatility over time—whether instability is rising or falling—rather than relying only on a universal “acceptable” percentage.
• Mutation testing can help uncover tests that miss defects. It provides evidence about whether tests detect particular behavioral changes, including when teams are concerned that tests may pass despite problems.
• Quality is a team responsibility. Testing should be integrated into development rather than treated solely as a separate phase or a specialist team’s job. Test specialists can still provide valuable coaching and support with test design, pipelines, infrastructure, reporting, and verifiable requirements.
• Use AI as an assistant, not an accountable owner. Dijkstra is comfortable using an LLM to help write product code when he writes and understands the tests himself. For testing work, he suggests using AI to review test code, analyze results, suggest test cases, or provide another perspective. He warns that AI can generate too many tests or make them appear reassuring without proving they check the intended behavior. People remain accountable for evaluating the output; mutation testing and code review can provide additional checks.
• Testing complex workflows presents extra challenges. Dijkstra describes work with a Dutch utility company using Temporal, a workflow-management platform. Integration tests for workflows involving timeouts, sequences, and long-running processes make mutation testing particularly technically demanding.

Guides, reviews, and examples mentioned

• Practical guidance: Ask what a metric actually tells the team; investigate test failures rather than retrying until green; monitor instability over time; and use AI for suggestions or analysis while retaining human responsibility.
• Example: Dijkstra used Claude to generate most of a vulnerable API for a security-testing workshop, then wrote the tests himself so he could verify the API’s behavior.
• Further reading: He mentions a detailed article about Facebook/Meta’s efforts to measure, improve, and predict test-suite volatility. No formal step-by-step tutorial is presented in the interview.

Main speakers

• Jenna Beckett — interviewer and Cypress host.
• Bas Dijkstra — test-automation consultant and CypressConf 2026 keynote speaker.