A Risk Based Approach to Testing

Summary

Karen Samby describes how Just Eat Takeaway.com uses a risk-based approach to testing within cross-functional Agile teams. The central idea is to match the level of quality-engineering involvement to the likelihood and impact of a problem, rather than having a quality engineer test every change.

How quality engineering has changed

• Samby’s roles have evolved from test technician and automation analyst to mobile and quality engineering roles.
• The shift from Waterfall to Agile has moved work from separate, specialized teams toward cross-functional teams that include developers, designers, and quality engineers.
• Testing and automation have become more integrated with development. Samby describes a move from predominantly “black-box” automation toward “white-box” testing that engineers can work with directly.
• Teams generally own particular areas of the app, while Samby’s Android platform team works across modules on topics such as Android APIs, libraries, CI pipelines, and releases.

Quality engineers’ role

With roughly five engineers per quality engineer, quality engineers cannot be the sole testers for every piece of work without slowing feedback and creating a bottleneck. Developers are expected to test their own work, with quality engineers supporting them through coaching, guidance, and documentation.

This approach lets quality engineers spend more time on complex or high-risk work, cross-team coordination, research, proof-of-concepts, and process improvements. The broader goal is to shift testing left: identify problems during development, ideally while a change is being made, rather than after release, when fixes may require hotfixes or harm users’ experience.

Risk-based testing process

Risk is treated as a combination of how likely an issue is and how serious its impact would be. The team uses risk assessment to decide how much testing and quality-engineering involvement a change needs.

1. Review upcoming work early. After receiving requirements and work streams, Samby and a product owner or technical manager hold a pre-refinement review. This usually happens weekly, often on Fridays, after the discovery phase.
2. Identify possible sources of risk. These may include:
   • Incomplete or unclear requirements
   • Dependencies on other teams or third parties
   • Limited time or the need to learn new technologies
   • Changes in particularly sensitive or important parts of the app
3. Use a risk scorecard. The scorecard assigns points to relevant risk areas. Its categories are a guide and should be adapted to each company. Especially high-concern areas include safety, allergens, legal requirements, and revenue. Less prominent areas may include parts of the app that are used less often or are less important to customers. The more points a change receives, the greater its assessed risk.
4. Assign a risk level and plan the response.
   • Low risk: A simple change, such as a text or button update, may need only straightforward developer testing.
   • Medium risk: The response varies by task and may involve coordination with other teams or additional review.
   • High risk: Quality engineering may provide detailed test and execution plans, coordinate with stakeholders, set up monitoring, and plan rollouts and feature flags.
5. Reassess when the work changes. The team revisits risk when a project’s scope changes, using regular pre-refinement sessions to update its assessment.

Risk information is recorded with work tickets so it can be categorized and filtered.

Testing practices and examples

• UI tests are developed as part of feature work, with developers and quality engineers sharing responsibility for the test suite.
• Tests run for each pull request, and changes cannot be merged if required tests are failing.
• The team does not aim for 100% automated coverage. It prioritizes critical scenarios, including allergen-related areas and key ordering flows.
• A third-party vendor runs a manual regression pack for releases, which occur every two weeks and cover areas that cannot be automated.
• Performance monitoring is an area the team is continuing to improve. Samby mentions using Macrobenchmark to help examine jank.
• Accessibility is checked during the manual release process and supported by an accessibility guild. The team uses Google-provided tools and pays attention to screen-reader support, including TalkBack and VoiceOver, as well as UI content descriptions.

Benefits of the approach

• Quality engineers can focus on work where their involvement adds the most value.
• Developers gain confidence and independence in testing their own changes.
• Testing begins earlier, helping the team find issues before they become release problems.
• The level of testing and planning is proportionate to the risk of the work.

Speakers and source

• Karen Samby — presenter and quality engineer at Just Eat Takeaway.com.
• Unidentified audience members — asked questions during the Q&A.
• Source: Novoda video, A Risk Based Approach to Testing.