GCI World 2026 September Session6 During Lecture

Model Evaluation

1. Supervised Learning and the Evaluation Workflow

Supervised learning estimates a relationship between input features (X) and a target (Y). The target determines the task:

• Regression predicts continuous values, such as prices or temperatures.
• Classification predicts discrete labels, such as spam/not spam or malignant/benign.

A reliable evaluation workflow separates data by purpose:

• Training set: Used to fit the model.
• Validation set: Used during development to compare models and tune hyperparameters.
• Test set: Reserved for a final evaluation on data the model has not seen.

The lecture’s workflow is to collect data, explore it, preprocess it, split it, train and evaluate models, tune them using validation results, and finally test the chosen model. Preprocessing should be applied consistently. The test set should not be used to choose hyperparameters, since doing so undermines its role as an independent check.

Evaluation is needed both to assess generalization—whether the model works on new data—and to choose hyperparameters such as regularization strength or tree depth. A model that memorizes its training data but performs poorly on new examples is overfit.

2. Classification Metrics

Confusion Matrix

A confusion matrix compares actual labels with predictions. With the positive class specified:

• True positive (TP): Actual positive, predicted positive.
• True negative (TN): Actual negative, predicted negative.
• False positive (FP): Actual negative, predicted positive.
• False negative (FN): Actual positive, predicted negative.

Rows commonly represent actual labels and columns predicted labels, though the convention should always be checked.

Main Metrics and When to Use Them

• Accuracy: ((TP + TN)/(TP + TN + FP + FN)) The proportion of all predictions that are correct. It can be misleading when classes are imbalanced: a model can score highly by mostly predicting the majority class while missing important minority cases.

• Recall (sensitivity): (TP/(TP + FN)) The proportion of actual positives detected. Use it when missing a positive case is especially costly—for example, cancer screening or fraud detection.

• Precision: (TP/(TP + FP)) The proportion of predicted positives that are truly positive. Use it when false alarms or follow-up actions are costly.

• F1-score: The harmonic mean of precision and recall, (2PR/(P+R)). It gives a single measure that is high only when both precision and recall are reasonably high.

The disease-screening example illustrates the danger of relying on accuracy alone. In a dataset with many healthy cases and very few sick cases, a model can achieve 99% accuracy while detecting only a small fraction of sick patients.

The positive-class definition matters. In the breast-cancer example, the labels are 0 for malignant and 1 for benign. A library that treats 1 as positive will report benign-class recall by default. To assess cancer detection, the positive label must be set to malignant. An actual malignant case predicted as benign is a false negative.

Thresholds, ROC Curves, and AUC

Many classifiers produce probabilities or scores, not just labels. A threshold converts a score into a positive or negative prediction. Changing the threshold changes precision and recall, so these metrics depend on both the model and the chosen threshold.

• An ROC curve plots the true positive rate (recall) against the false positive rate as the threshold varies.
• ROC AUC measures the area under that curve. It summarizes how well the model ranks positive cases above negative cases across thresholds. An AUC of 1 indicates perfect separation; 0.5 corresponds to random ranking.

The lecture describes constructing an ROC curve by calculating probabilities, trying thresholds, computing true and false positive rates for each threshold, and plotting the resulting points. In practice, scikit-learn can calculate the curve and AUC directly.

Choose metrics according to the consequences of errors: prioritize recall when misses are costly, precision when false positives are costly, F1 when both matter, and ROC AUC when comparing ranking ability across thresholds. Accuracy is most informative as a quick summary when the classes are reasonably balanced.

3. Classification Demonstrations

The breast-cancer notebook demonstrates how to:

• Split data into training, validation, and test sets while using stratification to preserve class proportions.
• Tune an SVM’s (C) and (\gamma) parameters by comparing validation accuracy across a grid of values.
• Refit the selected model and evaluate it on the reserved test set.
• Calculate a confusion matrix and derive accuracy, precision, recall, and F1, either manually or with scikit-learn.

A second demonstration uses predicted probabilities to show how different thresholds produce different labels and ROC points. The instructor emphasizes checking which class is considered positive before interpreting the metrics.

4. Regression Metrics

Regression predicts continuous values. Errors are the differences between actual values and predictions. Because errors can be positive or negative, simply averaging them can allow them to cancel out.

• Mean squared error (MSE): The average squared error. Squaring prevents cancellation and strongly penalizes large errors, making MSE sensitive to outliers.
• Root mean squared error (RMSE): The square root of MSE. It retains the strong penalty for large errors but is expressed in the target’s original units.
• Mean absolute error (MAE): The average absolute error. It is less sensitive to unusually large errors than MSE or RMSE.
• (R^2), the coefficient of determination: Compares the model with a baseline that always predicts the target’s mean. A value of 1 is a perfect fit; 0 is equivalent to the mean baseline; a negative value means the model performs worse than that baseline.

MSE and RMSE are useful when large mistakes should carry substantial cost. MAE is useful when a more outlier-resistant average error is preferred. (R^2) provides a scale-independent comparison with a simple baseline.

5. Regression Demonstration and Practical Lessons

The California housing example compares linear regression, ridge regression, a decision-tree regressor, and linear SVR. Features are scaled for models that are sensitive to feature scale, and the models are compared using multiple error metrics and (R^2).

The results illustrate why no single metric tells the whole story: the decision tree performs better on some absolute-error measures but worse on MSE and (R^2), suggesting that a few large errors affect its squared-error results. The linear SVR’s negative (R^2) indicates performance worse than predicting the mean. Predicted-versus-actual plots help reveal patterns and outliers that summary numbers may hide.

The lecture’s central lessons are:

1. Use multiple metrics rather than trusting a single score.
2. Choose metrics based on which errors matter most in the application.
3. Use visualizations alongside numerical measures.
4. For imbalanced classification, examine class-specific performance and ensure the positive class is defined correctly.
5. In a cost-sensitive problem, choose a model based on the real costs and benefits—not simply whichever model has the highest conventional accuracy.

The final bank-marketing exercise applies this last principle. If failing to contact someone who would accept an offer costs much more than making an unnecessary call, the evaluation should emphasize detecting likely acceptances. The model selected by that objective may differ from the one selected by accuracy and therefore may produce a different financial outcome.

Speakers and Sources Featured

• Recorded lecture presenter: Unnamed; presents the model-evaluation material and notebook demonstrations.
• Live-session instructor/commentator: Unnamed; discusses the Colab exercises, confusion-matrix interpretation, class labels, ROC/AUC, and the practical cost-based assignment.
• “Dave”: Mentioned by the live instructor as discussing important aspects of the class; the subtitles do not clarify whether this is the recorded presenter.
• Tools and materials referenced: Google Colab, scikit-learn, and the course notebook/IPYNB materials.
• Datasets and examples used: Breast-cancer classification, California housing regression, disease diagnosis, and a bank deposit-calling scenario.