Video summary

A/B-тесты в ML: как не ошибиться в результатах

Main summary

Key takeaways

Educational

Main ideas / lessons (A/B-testing for ML)

Why A/B-tests are needed

  • A team ships a recommendation (or ML) model, waits ~1 month, and metrics worsen.
  • A likely issue is that there was no preparation to correctly evaluate impact under changing conditions (e.g., seasonality).
  • A/B-tests help make data-driven decisions:
    • confirm or refute the hypothesis,
    • reduce the risk of drawing wrong conclusions after deployment.

Core concepts behind A-tests (3 pillars)

1) Randomized groups (A vs B)

  • The “A” and “B” must come from users assigned by chance (not sequential assignment like “first 1,000 users” to control and the next to test).
  • Mechanism to randomize assignment
    • Use a user ID (number/string).
    • Apply a hash function to map the ID to a fixed value.
    • Convert hash output into a number/bytes stream.
    • Split into groups by (example) percentage separation or effectively a bit from the hash.
  • Avoid bias across multiple tests
    • If you run many hypotheses/tests, you need different partitions each time.
    • Add a salt value to the hash input, e.g.:
      • task identifier (task manager number),
      • date or test ID.
    • Use hash(user_id + salt) so the partition is uniquely determined per test.

Goal: ensure group assignment is effectively random for each experiment.

2) Metrics history (what you measure)

  • Choose a metric such that changes reflect whether you’ll be happy or not.
  • Revenue/profit is hard to influence directly (e.g., you can’t just “repaint buttons” to control money precisely).
  • Therefore use proxy metrics that better capture usage/success of the service, such as:
    • conversions,
    • send/engagement times,
    • other product metrics.
  • Build conclusions about business impact through these proxy metrics.

3) Significance (is the observed difference real?)

  • Metrics fluctuate over time; observed changes may be noise.
  • P-value (misinterpreted often)
    • P-value = probability of getting the same or more extreme outcome.
    • It does not mean “your hypothesis is true.”
    • Interpreting it correctly:
      • Large P-value often means not enough data to conclude.
      • It does not necessarily mean there is no difference.

  • Two error types

    • Type I error (alpha): finding an effect when none exists. Example analogy: doctor says a man is pregnant. Controlled at a typical level like 5% (α).

    • Type II error (beta): missing a real effect. Example analogy: doctor says a clearly pregnant woman is not pregnant. Often at a typical level like 20% (β).

  • Trade-off:

    • You can’t reduce Type I and Type II simultaneously.
    • Balancing them is central to experimentation/analytics.

The key principle: A/B-test design happens BEFORE launch

“A-test is not what you do after, but what you do before”

  • Design stage records everything up front:
    • hypothesis,
    • metrics,
    • evaluation criteria,
    • error levels (alpha/beta),
    • expected effect size,
    • required sample size and duration constraints.

Why design is interconnected (multi-criteria optimization)

  • Smaller effect sizes require:
    • larger sample
    • longer run time
    • e.g., from 1 week to 2 weeks.
  • Business may not tolerate long tests.
  • Risk if you decide too quickly:
    • you may end up with “no effect confirmed,”
    • effectively wasting time and effort.

MDE: Minimal Detectable Effect

  • MDE = smallest change the design can reliably detect.
  • MDE can be computed:
    • analytically (formulas using dispersion + sample size),
    • or empirically via simulations.
  • Example problem:
    • If you want to detect 1%, design might require 3 weeks.
    • If business shortens to 1 week, MDE increases sharply
      • potentially from 1% to 20–30%,
      • making small true effects hard to detect.

If MDE becomes unacceptable, two mentioned options

  • Adjust expectations / shorten test carefully
    • Shortening increases MDE (often dramatically), so this works only if early “low-hanging fruit” effects are likely (rare).
  • Use metric engineering
    • Build tree metric / chain of proxy metrics
    • Aim to find a metric that changes faster (more sensitive), when you don’t have enough “divisions” / statistical power for the original metric.

How “correctly designed” tests still fail (3 traps)

Trap 1: Peeking / early stopping (“don’t stop too early”)

  • Scenario:
    • You planned to wait ~2 weeks.
    • After a few days the metric looks worse than your threshold.
    • Temptation: stop early and declare failure.
  • What can happen:
    • Early fluctuations made it look bad,
    • If you had stopped, you’d have rolled out the model incorrectly.
    • Even if early group performance looks decisive, significance may disappear by the end.
  • Consequence:
    • Premature stopping inflates Type I error
    • e.g., from intended 5% up to 15%/30%.
  • Correct practice:
    • Monitor for issues, but keep decisions consistent with the pre-set design.
    • Don’t repeatedly “check and stop” based on interim results.

Trap 2: Multiple comparisons (more tests ⇒ more false positives)

  • Example failures:
    • not 2 groups but 4,
    • not 1 metric but 2 metrics across 2 groups.
  • Effect:
    • Type I error increases with the number of comparisons.
    • With ~10 comparisons, Type I error can rise from 5% to ~40%.
  • Mitigations mentioned:
    • Baneroni / Holm-Bonferroni corrections
    • Trade-off: they reduce power (harder to detect real effects) or require more data.
  • Practical advice:
    • Avoid multiple hypothesis testing as much as possible.

Trap 3: Data issues

  • Always monitor for anomalies:
    • technical accounting changes,
    • unusual values,
    • preprocessing/pipeline failures.
  • Data preparation must be consistent:
    • the same preprocessing steps should be applied before and after the test to compute effects correctly.
  • If data is scarce:
    • consider variance reduction techniques.

Tools when data is limited

Variance reduction methods

  • Goal: reduce metric dispersion to detect smaller effects with fewer samples.
  • Kuped method
    • Use user behavior history before the test.
    • Train/predict user actions in the future.
    • Subtract a component independent of the metric mean, so:
      • the average stays similar
      • the dispersion decreases
    • Result: fewer required data/time (can reduce from “tens of weeks to ~1 week” in favorable cases).
  • Other related approaches (mentioned):
    • rank transformations,
    • stratification,
    • linearization.

Consistent/Sequential analysis (“don’t just peek; do it correctly”)

  • After discussing “no peeking,” the video introduces a safe alternative:
  • Wald method (invented 1945; described as a military-origin technique)
    • accumulate data step-by-step,
    • after each step compute likelihood,
    • stop only when boundaries are crossed.
  • Modern descendant mentioned:
    • “A/B testing that can be checked at least every day” (sequential/consistent testing style).
  • Claim:
    • Type I error remains controlled (relative to α).
  • Mentioned as course material:
    • dedicated lecture/seminar on implementation.

Takeaways (what viewers should remember)

  • A/B testing isn’t only the running of experiments; it’s the entire pre-launch design:
    • hypotheses, metrics, criteria, user grouping method, expected effect, data volume, timeline.
  • The design stage should include:
    • simulations,
    • studying problematic cases,
    • communicating with the business,
    • aligning requirements and acceptable trade-offs,
    • ensuring the experiment results will be reliable in practice.
  • In practice, many A/B tests (stated as 80% or more) may remain inconclusive/gray; this is normal.
  • The A-test’s purpose is to avoid costly mistakes and support a data-driven decision process you can trust.

Speakers / sources featured

  • Speaker: Daniil Potapov (Data Science expert)
  • Methods / referenced sources (not people):
    • P-value (mathematical statistics concept)
    • Wald method (invented 1945; military origin)
    • Benjamini/Fanferoni / Holm-Bonferroni (multiple comparison corrections—names as stated)
    • Kuped (interpreted as CUPED, mentioned in text as “Cupid/KupED”)
    • Mentions of Python code usage (implementation context in the course)

Original video