Lec 2: Performance Measures of Classification

Main ideas / lessons

• Performance evaluation in classification is done after training a model and using it to make predictions on validation/testing data.
• The core tool for evaluation is the Confusion Matrix, from which many metrics are derived.
• Key derived metrics include:
  • Accuracy
  • Precision
  • Recall
  • F1 score
  • (For ROC-based evaluation) ROC curve, TPR/FPR, and AUC
• The “best” metric depends on the application cost of errors:
  • Some tasks prioritize avoiding false negatives (Recall).
  • Others prioritize avoiding false positives (Precision).

---

Method / workflow for evaluating a classifier (step-by-step)

1. Train the model using a learning algorithm on a training set.
2. Perform hyperparameter tuning iteratively to improve performance.
3. Use a validation/testing set to evaluate performance:
   • Predict labels using the trained model.
   • Compare predictions vs. true labels.
   • Build a confusion matrix.
4. Use the confusion matrix to compute evaluation metrics.
5. Optionally compute ROC curve and AUC for threshold-independent comparison.
6. Stop training to reduce risk of overfitting.

---

Confusion Matrix: concept and how it’s used

• A confusion matrix is a table summarizing how well the classifier predicts examples across classes.

• It counts outcomes for each class:

  • True Positive (TP): predicted positive and actually positive
  • False Negative (FN): predicted negative but actually positive
  • False Positive (FP): predicted positive but actually negative
  • True Negative (TN): predicted negative and actually negative

• From TP/FP/FN/TN, compute performance metrics.

Examples described

• Binary (Email spam):

  • Actual: Spam / Not spam
  • Predicted: Spam / Not spam
  • Identify TP, FN, FP, TN accordingly.

• Binary (Cricket tournament outcomes):

  • True positive/negative and false positive/negative correspond to whether the predicted winner status matches actual results.

• Multi-class (Emotion classification):

  • Emotions: Happy, Sad, Angry, Surprise, Disgust, Neutral
  • The confusion matrix counts how often each emotion is correctly predicted and how often it is misclassified into other emotions.

---

Metrics derived from the confusion matrix

1) Accuracy

• Idea: fraction of correct predictions.

• Formula (as described): [ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} ]

• Interpretation: Higher accuracy = more correct classifications overall.

2) Precision

• Idea: among all predicted positives, how many are actually positive.

• Formula (as described): [ \text{Precision} = \frac{TP}{TP + FP} ]

• Interpretation: Precision is closely tied to the cost of false positives.

3) Recall (Sensitivity / TPR)

• Idea: among all actual positives, how many are correctly predicted positive.

• Formula (as described): [ \text{Recall} = \frac{TP}{TP + FN} ]

• Interpretation: Recall is closely tied to the cost of false negatives.

4) F1 score

• Idea: harmonic mean of Precision and Recall (balances both).

• Formula (as described): [ F1 = \frac{2}{\left(\frac{1}{\text{Precision}} + \frac{1}{\text{Recall}}\right)} ]

• Interpretation: F1 = 1 when Precision and Recall are both perfect.

5) (Visualization explanation)

• Precision corresponds to the portion of predicted positives that are correct:
  • ( \frac{TP}{TP + FP} )
• Recall corresponds to the portion of actual positives that are found:
  • ( \frac{TP}{TP + FN} )

---

Choosing the “best” metric depends on the problem

Cancer diagnosis example (prioritize not missing cancer)

• Missing a cancer case is very costly (false negatives).
• Recommended emphasis: Recall [ \text{Recall} = \frac{TP}{TP+FN} ]

Email spam example (prioritize catching spam)

• The priority is not missing important spam emails.
• Recommended emphasis: Precision [ \text{Precision} = \frac{TP}{TP+FP} ] (Note: the lecturer frames it as avoiding missed important spam; they still choose Precision in the narration.)

---

Multi-class metric computation guidance (as described)

• For each class:
  • Recall for a class uses the row (TP + FN).
  • Precision for a class uses the column (TP + FP).
• Accuracy in multi-class is based on diagonal entries (correct classifications).

---

ROC curve and AUC (threshold-based vs threshold-independent)

Definitions

• ROC = Receiver Operating Characteristics

• Uses:

  • True Positive Rate (TPR) = Recall [ TPR = \frac{TP}{TP+FN} ]

  • False Positive Rate (FPR) [ FPR = \frac{FP}{FP+TN} ]

  • Specificity (mentioned): [ \text{Specificity} = 1 - FPR ]

How ROC is formed

• The classifier decision uses a threshold.
• Different thresholds produce different (TPR, FPR) pairs.
• Plot these pairs to form the ROC curve.

AUC (Area Under the ROC Curve)

• AUC ranges from:
  • 0 = totally wrong predictions
  • 1 = perfect classifier
• Advantage emphasized:
  • AUC is threshold-invariant, making it useful for comparing classifiers.
• The lecturer also describes how to locate ROC points from confusion matrix values in examples.

---

Summary of what can be computed (as listed in the talk)

From confusion matrix:

• TP, TN, FP, FN
• Accuracy
• Precision
• Recall
• F1 score
• Specificity (via FPR)
• Misclassification rate (based on incorrect predictions, especially FN depending on framing)
• Rejection rate (example: alphabet recognition—rejects inputs not in the learned set)

From ROC:

• ROC curve (TPR vs FPR across thresholds)
• AUC (overall performance summary for comparison)

---

Speakers / sources featured

• No other speakers or named sources are explicitly identified.
• The only implied source is the lecture instructor delivering “Lec 2: Performance Measures of Classification.”