LECTURE 8

Summary

The lecture continues the discussion of pre-deep-learning machine-learning methods for Earth observation, focusing on support vector machines (SVMs) and decision trees. It explains how SVMs handle nonlinear and multiclass problems, describes an SVM-based domain-adaptation method, and introduces decision-tree splitting, model errors, and overfitting.

Nonlinear and multiclass SVMs

A linear SVM works when classes can be separated by a straight line or hyperplane. Soft margins can tolerate some misclassified or noisy examples, but they do not solve cases where the data is fundamentally nonlinearly separable.

One way to handle nonlinear data is to map it into a different feature space where a linear separator may work. The resulting boundary can still be nonlinear when viewed in the original space. Kernel methods make this approach practical: rather than explicitly transforming data into a high-dimensional space, kernel functions compute similarities that support separation there.

Standard SVMs are designed for binary classification. Two common strategies extend them to multiple classes:

• One-versus-rest: Train one classifier per class, treating that class as positive and all other classes as the rest. At prediction time, assign the sample to the class whose classifier gives the highest decision score.
• One-versus-one: Train a classifier for every pair of classes. With (n) classes, this requires (n(n-1)/2) classifiers. Each classifier votes for one of its two classes; the class with the most votes is selected.

SVMs remain useful in remote sensing, including hyperspectral classification, because they can work with relatively few samples, handle high-dimensional data, and have a strong mathematical foundation. They can also be used for regression.

The lecture gives a simple Python example using scikit-learn to train an SVM on labeled points and predict labels for new points.

SVMs for domain adaptation

The lecture describes an SVM-based domain-adaptation approach for situations where the source and target domains have different data distributions:

1. Begin with labeled samples from the source domain and train an SVM.
2. Apply the classifier to unlabeled target-domain samples.
3. Gradually incorporate target samples judged reliable, reducing the source data’s influence while increasing the target data’s influence.
4. Use circular validation to assess whether the adaptation is meaningful:
   • Forward adaptation: Adapt the classifier from the labeled source domain toward the unlabeled target domain.
   • Backward adaptation: Use the target predictions as pseudo-labels to adapt back toward the source domain.
   • Since source labels are available, evaluate the recovered classifier on source data. Good recovery of source-domain performance is taken as evidence that forward adaptation may have learned a useful target classifier.

Decision trees

A decision tree is a supervised-learning model that makes predictions by repeatedly splitting data according to feature values. Its structure consists of:

• Internal nodes, which specify decision rules;
• Branches, which represent possible outcomes of those rules;
• Leaf nodes, which provide predictions.

Decision trees are intuitive and easy to interpret. They can also represent nonlinear relationships.

A common way to choose a split is information gain:

• Entropy measures the uncertainty or impurity of a dataset.
• Information gain measures how much entropy is reduced by splitting on a feature.
• At each node, the tree selects the candidate split with the greatest information gain.

Error, underfitting, and overfitting

• Training error is the error on the training set.
• Test error is the error on the test set.
• Generalization error is the expected error on new data drawn from the same distribution.
• Underfitting often occurs when a model is too simple; both training and test errors tend to be high.
• Overfitting often occurs when a model is too complex; training error may be low while test error is high.

Potential causes of overfitting include:

• Too little training data;
• Training examples that do not represent the broader data population—for example, examples from only one part of a city may not generalize to the whole city;
• Excessive model complexity.

When two models have similar generalization performance, the lecture recommends preferring the simpler one. Decision trees are easy to understand and can capture nonlinear patterns, but they are prone to overfitting. This motivates the next lecture’s discussion of methods built on decision trees.

Speakers and sources featured

• Speaker: Unnamed course lecturer from the NPTEL course AI for Earth Observation.
• Sources and tools mentioned: Support vector machines; an SVM domain-adaptation study (not named in the subtitles); Python’s scikit-learn; ImageNet as an example of a large dataset; remote-sensing and hyperspectral-classification research.