MTS Your first recsys - 2

Overview

Lecture covering validation and baselines for recommender systems (part of a course). Main focus: how to set up realistic offline validation, which metrics to use, practical caveats, and simple baseline models.

Validation and deployment considerations

• Purpose of validation:
  • Check model quality and compliance with deployment requirements.
  • Validation should reproduce production usage as closely as possible.
• Important questions when designing validation:
  • Is the model producing scores or ranked lists?
  • Are recommendations built periodically (batch) or online (real‑time)?
  • What are constraints on training time and recommendation latency?
  • How should already‑seen items be treated (filter out watched content vs re‑recommend consumables like groceries)?
  • How severe is cold start (new users/items) in your product?

Validation should mimic production behavior and answer operational constraints (latency, batch vs online, re-recommendation policy, cold start).

Data partitioning (offline test splits)

Three common offline partitioning strategies:

1. Random partitioning (e.g., leave‑one‑out / leave‑k‑out)
   • Hide random interaction(s) per user/item for testing.
2. Chronological partitioning
   • Hide the latest interactions (globally or per user/item) using dataframe head/tail logic.
3. Time‑based split
   • Choose an interpretable test window (day/week/month) and split by timestamp.

Cold vs warm evaluation scenarios:

• Cold: test users/items have no interactions in training.
• Warm: users/items appear in training but have new interactions in the test window.

To simulate online cold‑start for incremental models:

• Hide a user entirely from the training set and evaluate recommendations for them in test.

Practical implementation notes

• Work at the user×item pair level: training/test and predictions are dataframes keyed by user and item (optionally with value columns like rating or watch duration).
• Combine test ground truth with recommendations using a left‑join to create a single evaluation dataframe.
• Useful dataframe operations: sampling, grouping, head/tail, filtering by date (typical pandas-like functions).
• Keep evaluation data in long form: rows with (user, item, rank, relevance) are convenient for metric computation and vectorization.

Metric families and usage

1. Regression metrics

   • For score prediction (e.g., rating prediction): MSE, MAE, RMSE.
   • Compute per user×item and average.

2. Classification metrics (treat recommendations as binary: recommended / not recommended)

   • Compute TP, FP, FN, TN per user (confusion matrix).
   • Commonly used: precision and recall (especially precision@k and recall@k).
   • Note: precision tends to be small in large catalogs when recommending top‑K because relevant interactions are sparse.
   • Typical calculation approach:
     • Mark recommended items (with position/rank).
     • Restrict to top‑k.
     • Aggregate per user then average across users.

3. Ranking metrics (weigh positions of hits)

   • MRR (mean reciprocal rank / average inverse rank)
     • For each user take the reciprocal of the position of the first relevant item; average across users.
     • Useful when each user has 1–2 interactions in test.
   • MAP (mean average precision)
     • Average precision at relevant positions per user, then average across users.
     • Useful when users may have multiple relevant items.
   • DCG / nDCG (discounted cumulative gain and normalized DCG)
     • Assign position weights (discounts), sum gains up to min(k, #relevant), normalize by ideal DCG to get nDCG.
     • Useful to evaluate ranking quality across positions.

• Metrics are computed from the joined long‑form dataframe (user, item, rank, relevance) and can be vectorized for efficiency.

Caveats & interpretation

• Precision/recall-like metrics ignore true negatives due to very large catalogs, which is acceptable for recommender evaluation.
• Choice of k (top‑K) matters; metrics should be normalized appropriately (per user) to avoid bias from users with very different interaction counts.
• Always ensure offline split and metric choices reflect how recommendations will be used in production.

Baselines — popularity model

• The most common baseline is the popularity (most‑popular) recommender:
  • Rank items by frequency of occurrence in training data and recommend top items to users.
• Variants and improvements:
  • Time‑windowed popularity (e.g., last 30 days) to capture recency/trends.
  • Popularity conditioned on user segments (e.g., by age group), item categories (genre), or context (next‑item popularity given previous item).
  • Next‑item popularity using sequential patterns when domain has clear purchase sequences.
• Benefits:
  • Simple, fast, and provides a useful floor to compare more complex models against.
• Implementation outline:
  • Filter interactions by history depth, count item frequencies, sort and return top items (possibly repeated per user).

Resources / practical exercises

• A Colab / notebook (“laptop”) with example code is available to practice the metrics and baseline implementation.
• Course channels: instructor offered to answer questions via private message or the course Telegram chat.

Main speaker / source

• Daniil Potapov (lecture from the “Your first recsys” course).