LECTURE 9

Summary

The lecture introduces ensemble learning, focusing on random forests built from decision trees. It then covers the fundamentals of perceptrons and multilayer perceptrons (MLPs) as background for later deep-learning topics.

1. Ensemble learning and random forests

A single decision tree can overfit its training data. Ensemble learning addresses this by combining predictions from many models rather than relying on one. The lecture describes this as the “wisdom of crowds”: a group of reasonably good models can produce more reliable results than a single model, especially when their errors differ.

A random forest is a collection of decision trees that are deliberately made different so their predictions can complement one another.

How a random forest is built

1. For each tree, create a bootstrap sample by randomly sampling the training examples with replacement. Some examples may appear multiple times, while others may be left out.
2. At each node, randomly select a subset of the available features.
3. Search for the best split using only that feature subset.
4. Repeat the process to grow many trees.
5. Combine the trees’ predictions, typically by voting for classification.

Random feature selection helps prevent a small number of dominant features from controlling every tree, encouraging the trees to differ. Because the trees make different mistakes, combining their predictions can reduce variance and improve generalization.

Advantages: Random forests can resist overfitting, handle nonlinear relationships, require relatively little preprocessing, and be robust to noisy data and outliers.

Limitations: They are less interpretable than a single decision tree, and very large forests require more computation.

2. Random forests in Earth observation

Random forests were widely used in Earth-observation work before deep learning became dominant and remain useful in some applications.

Two examples are discussed:

• A 2006 study classified a mountainous area in Colorado into 10 classes using Landsat spectral bands, elevation, slope, and aspect data. Variable-importance analysis found that Landsat band 4 was the most important feature, followed by band 2 and elevation. The authors reported satisfactory performance despite many outliers in the dataset.
• A 2005 example applied random forests to hyperspectral image analysis.

The lecture also highlights random forests’ robustness to geographic variation, attributing this in part to feature sampling and the ensemble approach.

The lecturer notes that, in their own observation—not presented as empirical evidence—random forests remain popular in some specific applications, including work with atmospheric data.

3. From perceptrons to multilayer neural networks

A perceptron takes multiple inputs, forms a weighted sum that includes a bias term, and produces an output. The original form uses a step function, but a simple perceptron cannot solve problems that are not linearly separable.

An MLP (multilayer perceptron) extends this idea by using multiple neurons and layers:

• Input layer: Receives the input features.
• Hidden layer or layers: Transform the features and can learn useful combinations and nonlinear relationships.
• Output layer: Produces the prediction. One output neuron may suffice for binary classification; multiple output neurons are used for multiclass classification.

Neurons in one layer are usually connected to every neuron in the next layer, so MLPs are also called fully connected neural networks. Unlike the original perceptron, MLP neurons use activation functions to introduce nonlinearity. The lecture mentions sigmoid and ReLU, noting that ReLU is widely used in CNNs.

4. Neural-network training and regularization

The training cycle for an MLP—and later, CNNs and vision transformers—consists of:

1. Forward propagation: Compute predictions from the inputs.
2. Loss computation: Measure prediction error.
3. Backpropagation: Calculate gradients of the error.
4. Weight updating: Adjust the network’s weights to reduce future error.
5. Repeat the cycle throughout training.

Larger, more complex networks can overfit by memorizing the training data. The lecture lists several ways to limit this:

• Use more training data.
• Apply data augmentation.
• Use dropout.
• Use early stopping.
• Apply L2 regularization.
• Tune network design and training choices, such as the number of layers, initial weights, activation function, and learning process.

The lecture distinguishes model parameters—internal values learned from training data, such as network weights—from hyperparameters, which are settings chosen by the user or operator.

Speakers and sources featured

• Speaker: An unnamed course instructor; no other speakers appear in the subtitles.
• Research examples mentioned: A 2006 study of mountain-area classification in Colorado and a 2005 hyperspectral-image-analysis example. The subtitles do not provide author names or full citations.
• Data source mentioned: Landsat spectral-band data, used in the Colorado classification example.