MTS Your first recsys - 1

Purpose and structure

Short practical-focused mini-course that collects the minimum necessary knowledge about recommender systems in one place. Theory links and additional reading are provided separately.

Lecture plan (high level):

1. Introduction to recommender systems: types, required data, high-level concepts (this lecture).
2. Evaluation metrics and simple baseline models.
3. Practical implementations using popular libraries — hands-on coding and comparisons.
4. Applying gradient-based methods (subtitle said “gradient Busting” — likely gradient boosting or other gradient methods) to recommendation tasks.
5. Advanced metrics, production integration, and final topics (review of popular approaches and advanced evaluation).

Use cases and motivation

Common real-world applications:

• Video personalization (YouTube).
• Movie recommendations (Netflix).
• Product recommendations (Amazon).
• Social networks (VK, Odnoklassniki, LinkedIn, Facebook).
• Search result ranking.
• Taxi/destination suggestions.

Historical case: the Netflix Prize

The Netflix Prize attracted research and popularized matrix factorization / hidden-representation models. Lessons: competition-winning complexity doesn’t always translate to production, and the metric choice matters (Netflix used RMSE, while ranking metrics are often more appropriate for deployed recommenders).

Core concepts and problem statement

• Main objectives:
  • Personal ranking: rank objects for a user by relevance (most common in practice).
  • Predict ratings (numerical or binary); sometimes used as features for downstream models.
• Actors: users and objects (movies, products, tracks, other users).
• Data representation: rating/interaction matrix (users = rows, items = columns). These matrices are extremely sparse (often well below 1% filled).
• Feedback types:
  • Explicit feedback: ratings, purchases (strong signal).
  • Implicit feedback: clicks, watch time, dwell (weaker but common). These can be combined with different weights.

Minimum dataset and engineering tips

Minimum data required:

• User IDs, object IDs, and interactions (click/purchase/timestamp).
• Often also include rating values and metadata.

Practical tips for working with large datasets (Pandas):

• Convert string ID columns to categorical dtype to reduce memory.
• Use integer dtypes when appropriate (avoid automatic float conversion for integer-like columns).

These two tricks can substantially reduce RAM usage (speaker claims up to ~80%), enabling larger datasets to be loaded on a single machine.

Sparse matrices and storage

Recommender libraries typically use sparse matrix types (SciPy sparse referenced). Common formats:

• COO (coordinate): stores row indices, column indices, and values. Good for construction.
• CSR / CSC (compressed sparse row / column): optimized for computations and storage. Use CSR/CSC for efficient arithmetic and row/column slicing.

Choose formats depending on whether you are creating the matrix or performing computations.

Features and data modeling

• Use both user and item metadata.
• User/item IDs can be represented as indicator (one-hot) features.
• Typical dataset layout: indicator features on one side, metadata (numerical/categorical) on the other.

What to recommend (strategies)

Non-personalized approaches:

• Global popularity.
• Category-level popularity.
• Popularity within demographic groups.

Simple personalized heuristics:

• Items similar to the last seen/purchased.
• Items similar to the recent N items.
• Items interacted with by similar users.

Main taxonomy covered:

• Non-personalized vs personalized.
• Personalized methods: content-based, collaborative filtering (neighborhood-based and model-based), and hybrid methods.

Course focus: collaborative filtering and hybrid approaches, as these are commonly used in production.

Collaborative filtering approaches

Neighborhood (k-NN) based:

• Treat rows (users) or columns (items) of the rating matrix as vectors.
• Build similarity matrices (user-user or item-item) and recommend based on nearest neighbors.

Model-based (matrix factorization / latent representations):

• Reformulate recommendation as missing-value recovery: approximate the rating matrix as the product of user and item latent-factor matrices.
• Latent vectors represent users/items; dot-product (or other similarity) yields predicted relevance.

These methods and their trade-offs will be implemented and compared in later lectures.

Evaluation and metrics

• Distinguish between rating accuracy metrics (e.g., RMSE used in the Netflix Prize) and ranking metrics (more appropriate for production ranking tasks).
• Historical lesson: optimizing RMSE doesn’t necessarily produce better ranked lists for users — prefer ranking metrics when the end goal is recommendation lists.

Resources, further reading and community

• Speaker provides links to channels, books, conferences, and competitions on recommender systems.
• Recommended to follow conference proceedings and recent competition winners to see current practical approaches.
• Practical materials and code will be provided in later lectures.

Practical promises in the course

• Hands-on implementations starting in lecture 3.
• Discussion of pros/cons of methods, advanced metrics, and production integration in later sessions.

Speakers and sources

• Main speaker: Daniil Potapov — head of the Recommender Systems & Personalization group at MTS BigData.
• Examples and sources referenced: Netflix Prize, and services such as YouTube, Netflix, Amazon, and social networks (VK, Odnoklassniki, LinkedIn, Facebook).