6 Evaluation

Main ideas / lessons conveyed

• Evaluation metrics vs. objective functions

  • Evaluation produces quantified scores about practical performance you care about in the application.
  • The objective function is mainly designed to guide training/optimization and may be aligned with but not identical to evaluation.
  • Good training loss/learning does not guarantee good real-world performance—you must check evaluation results.

• Inspect learning dynamics, not just one “best” point

  • Use validation/test performance traces over time (training/validation curves).
  • Selecting only one best validation point can be unstable and misleading because performance can vary substantially across runs/splits.

• Validate that your validation setup is proper

  • If your validation set is not representative, the chosen model may not generalize.
  • A key technique discussed: cross-validation.

• Choose metrics that match the problem type and its pitfalls

  • Classification: accuracy is common but can be misleading under class imbalance.
  • Regression: common metrics measure error magnitude and variance-related effects.
  • NLP: evaluation can be complex (e.g., fluency/quality of translation), so metric meaning may be less straightforward.

• Non-determinism requires repeated experiments

  • Many ML systems are non-deterministic (random seeds, stochastic optimization).
  • Evaluation should use multiple runs, reporting mean and variance (and checking significance).

• Cross-validation for stability

  • Cross-validation estimates performance robustly across different training/validation splits.
  • Proper selection of fold count (n) matters: too large or too small can increase variance or reduce coverage.

• Multi-objective optimization evaluation

  • In multi-objective problems, you evaluate sets of solutions via the Pareto/front idea (dominance), not a single scalar optimum.
  • A solution can be “best” only in some trade-off sense; dominated solutions are excluded from the front.

---

Key concepts and methodologies (detailed)

1) Designing/evaluating evaluation metrics: what to consider

• Ensure the metric reflects the real property you want from model outputs.
• Recognize that:
  • Training objective may optimize a proxy or surrogate loss.
  • Evaluation metric reflects what matters in practice.
• Don’t rely on a single metric—one metric won’t capture all aspects of performance.

2) Practical model selection (what to check beyond a single score)

• First check optimization behavior

  • Use training curves to verify the optimization/learning process is working (e.g., loss decreasing appropriately).

• Then check practical performance periodically

  • Evaluate on validation/test repeatedly rather than choosing one point.

• Be mindful of evaluation cost

  • Test set evaluation can be expensive (large test set), so full frequent evaluation may be limited, but tracing/monitoring is still emphasized.

3) Classification metrics

A) Accuracy

• Definition (conceptual): number of correct predictions / number of all predictions.
• Binary vs. multi-class: accuracy behavior depends on class structure.

B) Why accuracy can fail: class imbalance

• If one class dominates (e.g., 1% positive, 99% negative):
  • A naive model predicting the majority class can achieve very high accuracy while being useless for detecting rare cases.
• This problem is especially important in contexts like anomaly detection.

C) Precision and Recall (with truth table logic)

• Truth table categories (binary classification):

  • TP (True Positive): predicted positive, actually positive
  • FP (False Positive): predicted positive, actually negative
  • TN (True Negative): predicted negative, actually negative
  • FN (False Negative): predicted negative, actually positive

• Precision

  • Formula: precision = TP / (TP + FP)
  • Meaning: accuracy of positive predictions (how reliable predicted positives are).

• Recall

  • Formula: recall = TP / (TP + FN)
  • Meaning: coverage of actual positives (how many real positives are found).

• Conceptual distinction

  • Precision: “of what we predicted positive, how many were truly positive?”
  • Recall: “of all truly positive cases, how many did we capture?”

• Mentioned variants: precision@k, recall@k (precision/recall at a cutoff)

D) F1 measure

• Purpose: combine precision and recall into one metric.
• Formula: F1 = 2 * (precision * recall) / (precision + recall)
• Interpretation:
  • High F1 requires both precision and recall to be reasonably high.
  • If precision=1 and recall=0 (or vice versa), F1 becomes near 0.
• Caveat (implied): if the application prioritizes precision or recall separately, F1 may not match the actual goal.

E) ROC curve and AUC

• ROC curve: plots
  • x-axis: false positive rate (FPR)
  • y-axis: true positive rate (TPR)
• When emphasized:
  • When performance is unstable across noise/random seed or hyperparameter choices.
• Random classifier baseline:
  • If a classifier is random, performance clusters near the diagonal (or the corresponding random operating point).
• AUC (Area Under the ROC Curve):
  • Summarizes overall performance across thresholds.
  • Higher AUC implies superiority over baseline more generally across hyperparameter settings.

F) Multi-class classification

• Multi-class is more robust to imbalance than binary (though imbalance still matters).
• Mean precision approach:
  • Compute precision per class then average.
• Sampling/distribution bias issue:
  • Uniform sampling can be biased if the real-world class distribution is not uniform.
  • Rebalancing or assuming uniform can hurt general performance on other classes if it distorts training/evaluation assumptions.
• Key decision: follow natural training distribution vs. assume uniform depends on the scenario.

4) Regression metrics

A) MAE (Mean Absolute Error)

• Uses the average of absolute differences between predicted and true values.
• Concept: less sensitive to extreme outliers than squared error.

B) MSE / RMSE (Mean Squared Error / Root MSE)

• MSE: average of squared differences.
• RMSE: square root of MSE (used to control scale).
• Concept: squared errors penalize large deviations strongly (outliers get more weight).

C) R² (Coefficient of Determination)

• Described as reflecting how well predictions explain variance.
• Compares prediction error/variance relative to total variance (including a variance-based denominator).
• Purpose: indicate whether poor performance is due to model limitations or unavoidable variance.

D) Why multiple regression metrics are recommended

• Metrics (MAE, MSE, R²) emphasize different aspects (scale sensitivity, outliers, variance).
• A single metric may not describe performance from all perspectives, so evaluate multiple.

5) Validation / dataset splitting: cross-validation

Cross-validation procedure described (high-level)

• Split dataset into multiple folds.
• For each fold:
  • Use one fold as test/validation (validation/test swapping is discussed in the narrative)
  • Use remaining folds as training
• Repeat across folds and average performance.

Purpose

• Estimate stable/robust performance not overly dependent on a single train/test split.

Choosing the number of folds (n): trade-offs

• Large n
  • Smaller test sets → training/test similarity drops → higher variance and potentially lower average performance.
• Small n
  • Smaller training sets → insufficient coverage of patterns → lower generalization confidence.
• Core point: overlap between train and test distribution coverage matters, but it’s hard to know beforehand.

6) Evaluating non-deterministic models (randomness, seeds, variance)

• For stochastic training (random noise, randomized algorithms):
  • Performance varies across runs with different seeds.
• Recommended evaluation approach (described):
  • Run training multiple times with different random seeds.
  • Report average (mean) and variance.
  • Check statistical significance indirectly via variance (large variance → results may be insignificant).
• Reproducibility warning (seed handling):
  • Log/record the exact seed list/configuration.
  • Avoid uncontrolled random seed generation if you need repeatable research.

7) Multi-objective optimization evaluation (Pareto/front)

• Multi-objective: multiple objective functions where trade-offs exist and weights are unknown.
• Front / Pareto front:
  • Solutions on the front are non-dominated: no other solution is better in all objectives simultaneously.
• Dominance concept:
  • If one model has higher/better scores in all objectives compared to another, it dominates it.
  • Dominated solutions are not part of the Pareto front.
• Evaluation goal: compare which models appear on/near the front across different trade-offs.

---

Speakers / sources featured

• Speaker: One unnamed presenter (the narrator of the video).
• Sources mentioned: Wikipedia (recommended for browsing additional regression metric variants).