Сдал ML собеседование на 5+ в топовую Отечественную бигтех компанию

Main ideas / lessons from the interview

The video is an ML theory interview, with the discussion moving from classical ML to deep learning, and then to NLP. The interviewee demonstrates knowledge of:

• Core classical ML: linear regression, optimization via gradient descent, gradient descent variants
• Regularization concepts: L1/L2, later also dropout/batch normalization
• Ensembles: bagging/bootstrapping, random forests, and the bias–variance tradeoff
• Boosting: the general idea via gradient descent in prediction space; differences among popular GBM implementations
• Metrics for regression and classification, including handling outliers and precision/recall tradeoffs (ROC-AUC, PR curve ideas)
• Deep learning stabilization via dropout and batch normalization (including train-time vs inference-time behavior)
• Inference acceleration: quantization and optimized serving/framework tooling; mentions of model families like YOLO and transformers
• NLP representations and transformer internals: TF-IDF, embeddings (word2vec/FastText-like ideas), RNN vs transformers, and CLS pooling
• Transformer mechanics: positional encodings; encoder vs decoder; attention/FFN blocks; masking
• Practical training trick: gradient accumulation to emulate large batch sizes with limited memory

---

Methodologies / instruction-like content

1) Linear regression: problem, objective, and solution approach

• Definition: linear regression is a linear model mapping features to a real-valued target.
• Training objective: minimize MSE (mean squared error).
• Solution routes mentioned:
  • Analytical solution exists in principle.
  • Gradient descent can solve it iteratively.
  • Why not always analytical:
    • Requires computing the inverse of a matrix.
    • Matrix inversion is computationally expensive (cubic complexity cited).
    • The matrix may be non-invertible.

---

2) Gradient descent and its modifications

Adam

• Tracks first-order gradients (momentum-like behavior).
• Tracks second-order information (squared gradients).
• Updates parameters using a step direction derived from accumulated statistics.
• Normalizes steps by the accumulated second-moment estimate.

Stochastic / mini-batch gradient descent

• Updates parameters using subsets of the data rather than the full dataset.

Momentum

• Accumulates past gradients/updates.
• Uses an “accumulated gradient” rather than only the current batch gradient.
• Benefits cited:
  • More stable convergence
  • Often better convergence properties (brief reference to convex behavior)

---

3) Regularization (classical models)

• Goal: reduce overfitting by “dumbing down” the model.
• L1/L2 regularization for linear models:
  • Add a weight penalty to the loss function (penalty on weights).
  • Interpretation: limits weight magnitudes so no single feature dominates.
• Qualitative L1 vs L2 behavior (as described):
  • L1: “rhombus” geometry → more chance of hitting corners → exactly zero weights (sparsity).
  • L2: “circle” geometry → more distributed influence → less sparsity.
• Bias handling
  • Bias is typically not regularized, since it shifts the prediction baseline rather than controlling feature importance.

---

4) Bagging / random forest building logic

Bagging (bootstrap aggregation)

• Train multiple “independent experts” on different resampled datasets.
• Average predictions to reduce variance.

Random forest feature selection

Two ways were mentioned conceptually:

• Each tree uses a subset of features.
• Or features are randomly subsampled at each split/node.

Why feature subsampling?

• If many features are noisy or preprocessing introduced junk, random forests reduce reliance on junk features.

Bias–variance takeaway

• Individual trees can have higher variance.
• Averaging over many experts reduces variance roughly by dividing by the number of experts (conceptually described as variance ~ /N).

---

5) Boosting: general algorithm concept

• Boosting is an ensemble method, but built differently than bagging.
• Core idea described:
  • Perform gradient descent in the space of predictions.
  • Start with an initial prediction.
  • Compute the gradient of the loss (e.g., MSE) with respect to predictions.
  • Train a new weak learner to predict the negative gradient / residual (anti-gradient).
  • Add learners sequentially to reduce training loss.

---

6) Gradient boosting implementation differences (high-level)

• Libraries/models mentioned:
  • LightGBM, CatBoost (plus a general remark about XGBoost/others)
• Differences discussed:
  • Tree construction / depth handling (e.g., “symmetric tree” vs depthwise behavior)
• CatBoost-specific notes:
  • Target encoding support
  • A time-related parameter idea to prevent leakage (avoid looking ahead)

---

7) Choosing an algorithm under speed constraints

• When comparing Random Forest vs Gradient Boosting:
  • Both can be parallelized across trees conceptually.
• Speed depends on:
  • number of trees
  • tree depth
• Claim made:
  • With many shallow trees, random forests may benefit from horizontal parallelism.
  • Boosting could be faster depending on library optimizations and architecture (final assessment: depends).

---

8) Metrics: regression

• Metrics mentioned:
  • MSE, MAE, MAPE
• Handling outliers:
  • If outliers are common, prefer MAE, since it doesn’t square errors (unlike MSE).

---

9) Metrics: classification

• Confusion matrix components referenced:

  • TP, FP, FN, TN (only some explicitly expanded; TP/FP used for precision/recall)

• Metrics mentioned:

  • Accuracy: correct predictions / total samples
  • Recall: TP / (number of actual positives)
  • Precision: TP / (number of predicted positives)
  • F-beta score: harmonic combination of precision and recall; β controls weighting (recall discussed as “completeness”)
  • ROC curve / AUC: true positive rate vs false positive rate

• Practical examples:

  • Spam filtering: prioritize precision (avoid misclassifying real emails as spam)
  • Medical diagnosis (cancer): prioritize recall (avoid missing true positives)

---

10) Deep learning regularization: dropout and batch normalization

Dropout

• During training:
  • randomly “resets” some activations (sets them to zero),
  • scales remaining activations by a coefficient to preserve expected value.
• During inference:
  • dropout is not applied.
• Scaling behavior explicitly addressed:
  • if training reduces outputs (due to dropout probability), inference needs scaling correction so expected activations match.

Batch normalization

• During training:
  • normalize activations using batch mean/variance,
  • include trainable shift/scale parameters.
• During inference:
  • use running estimates accumulated during training (not current batch stats).
• Update behavior:
  • running averages updated with momentum-like weighting (example style: ~0.9/0.1).

---

11) Inference acceleration methods (high-level)

• Mentioned:
  • Quantization and deployment frameworks
  • Hardware/framework acceleration tooling (a term was mentioned as distorted, but the intent was optimized deployment)
  • Architecture choices efficient on devices (e.g., YOLO, transformers)
• General takeaway:
  • “Quantization and various components” for broad acceleration.

---

12) NLP: ways to create text / word embeddings (methods mentioned)

• TF-IDF
  • Sparse vectors using term frequency and inverse document frequency.
• FastText
  • Uses subword / n-gram information to better handle unseen words.
• Word embeddings via prediction (word2vec-like)
  • Two-matrix view mentioned; latent space translated toward dictionary-like representations.
  • Involvement of n-grams and predicting masked/middle elements (as described).
• RNN language modeling / embeddings
  • Hidden state carries information from previous tokens.
  • Hidden state updates and generates predictions.
• Transformers
  • Sentence-level vectors via a CLS token or pooling over token embeddings.

---

13) Transformer positional encodings

• Problem:
  • Attention alone is permutation-invariant → token order information is lost.
• Solution:
  • Add positional encoding so the model knows each token’s position.
• Types mentioned:
  • Absolute positional encodings (sinusoidal/cosine and trainable sine-like variants)
  • Relative positional encoding (mentioned generally)
• Integration note:
  • positional information is not only at the input, but also integrated into the model’s internal flow (as described).

---

14) Encoder vs decoder in transformers

• Common structure:
  • Multi-head attention + residual connections + normalization
  • then a feed-forward network (FFN) (two linear layers with activation)
• Decoder-specific elements:
  • Causal mask (prevents attending to future tokens)
  • Cross-attention:
    • keys/values (K/V) from the encoder outputs
    • queries (Q) from the decoder state

---

15) Practical training trick: gradient accumulation (large effective batch size)

• Problem:
  • GPU memory limits batch size (example: only batch size 32 fits; target is 2048).
• Method:
  • Accumulate gradients over multiple forward/backward passes:
    • run multiple mini-batches (examples like 32 → 64 were mentioned as intermediate steps),
    • don’t update the optimizer each step,
    • at the end, apply one optimizer update.
  • Scale gradients:
    • divide by the number of accumulation steps so the effective update matches a larger batch.
• Result:
  • achieves the same optimizer effect as if using a very large batch, while respecting memory limits.

---

Speakers / sources featured

• Interviewer: unnamed (as shown in subtitles)
• Interviewee: Mark (named in the closing: “Thank you for the interview, Mark”)