LECTURE 12

Overview

The lecture introduces supervised learning and traces the development of convolutional neural network (CNN) architectures. It explains how different design ideas address challenges in image recognition and remote-sensing classification.

Supervised Learning

Supervised learning consists of four components:

• Training data: Inputs, such as images or extracted image features, paired with targets, labels, or other feedback. Obtaining reliable labels can be challenging.
• Learner: The model, represented by parameters (θ).
• Learning algorithm: Updates the learner’s parameters to improve its predictions.
• Performance indicator: A cost or loss function used to assess the model’s performance.

The learning process can be viewed as an iterative cycle: predict, score, and learn.

The lecture distinguishes two approaches to image classification:

1. Extract features first, then train a classifier on them.
2. Learn the image features and classifier together.

Whether the second approach is preferable depends on factors such as the amount of training data available.

CNN Architectures and Their Development

LeNet-5

LeNet-5, developed around 1998, was an early CNN designed for small, 32 × 32 images. Its feature maps progress through the following layers:

• A convolution that produces 28 × 28 × 6 maps using 5 × 5 filters, stride 1, and no padding.
• A subsampling layer that produces 14 × 14 × 6 maps.
• Another convolution that produces 10 × 10 × 16 maps. This layer uses partial connectivity, so different output maps connect to different subsets of the input maps.

The lecture notes that the first convolution has 156 parameters, including biases. LeNet-5 had relatively few layers compared with modern CNNs.

The Pre-CNN Period in Remote Sensing

From roughly 2001 to 2010, remote-sensing classification methods often relied on engineered features followed by classifiers such as support vector machines (SVMs) or random forests. The lecture attributes the later growth of deep CNNs partly to improvements in GPUs and the availability of larger datasets.

ImageNet and AlexNet

The 2010 ImageNet Large Scale Visual Recognition Challenge introduced a large dataset with 1,000 object categories. Early leading approaches still commonly used feature extraction followed by SVMs, but CNNs subsequently performed better and helped popularize the approach.

AlexNet (2011–12) was a landmark CNN. Its notable features included:

• Five convolutional layers and three fully connected layers.
• A first convolutional layer using 96 filters of size 11 × 11.
• A design split across two GPUs, reflecting limited GPU memory at the time.
• ReLU activations, max pooling, and dropout.
• Extensive data augmentation using transformations that preserve an image’s class to reduce overfitting.

Dropout randomly disables some hidden units during training. This encourages the remaining units to learn useful features without relying too heavily on one another.

VGG

VGG (2014) emphasized repeated 3 × 3 filters and increased network depth rather than using large initial filters. The lecture presents this as an argument for building deeper networks with a consistent filter size. VGG had more layers and parameters than earlier models, while pre-initialized weights also contributed to its training behavior.

Inception Modules

Inception modules explore a different design direction from simply increasing depth. They process information using convolutions of different filter sizes to capture multiscale features. The lecture notes that this is particularly relevant to tasks such as semantic segmentation.

A 1 × 1 convolution is also shown as part of the module, though its purpose is deferred to a later discussion.

ResNet and Residual Connections

Simply adding layers does not guarantee better performance: deeper networks can perform worse, even when extra layers could theoretically implement identity mappings.

ResNet addresses this issue with residual blocks, which add a block’s input back to its transformed output. This helps deeper networks avoid the degradation that can occur when layers are added without such a mechanism.

The lecture closes by noting that the discussion of CNN architectures and learning paradigms will continue in the next lecture.

Speakers and Sources

• Speaker: One unnamed lecturer; no other speakers are identifiable in the subtitles.
• Architectures and research developments discussed: LeNet-5, AlexNet, VGG, Inception modules, and ResNet.
• Other sources and examples mentioned: The ImageNet Large Scale Visual Recognition Challenge, a Google Scholar search on remote-sensing classification papers, random forests, and SVMs.