LECTURE 13

Summary

The lecture reviews key CNN architectures and introduces transfer-learning options for situations where labeled training data is limited.

Residual Blocks and ResNet

Although a deeper network should theoretically be able to match or improve on a shallower network’s training error, optimization can make this difficult in practice.

A residual block adds its input to the block’s learned transformation:

[ Z_1 = F_1(X) + X ]

[ Z_2 = F_2(Z_1) + Z_1 ]

The block therefore learns a residual—the difference between the desired output and the identity mapping—instead of learning the entire mapping directly.

A basic implementation saves the input as an identity, processes it through convolution and post-processing layers such as batch normalization and ReLU, adds the saved input back, and applies a final ReLU. Skip connections are central to ResNet and are also used in many later architectures.

DenseNet

DenseNet connects layers densely within blocks: each layer receives feature maps from every preceding layer in that block, typically by concatenating them. A layer’s composite function may include batch normalization, ReLU, and a (3 \times 3) convolution.

If each layer contributes (K) feature maps, the number of input feature maps grows with depth. For layer (L), it is described as:

[ K_0 + K(L-1) ]

where (K_0) is the initial channel count. (K) is called the growth rate. DenseNet commonly uses a relatively small value, such as 12, to limit the growth in feature maps.

Dense blocks are separated by transition layers, generally consisting of batch normalization, a (1 \times 1) convolution, and (2 \times 2) average pooling. These layers control channel growth and reduce spatial feature-map size.

MobileNet and Efficient Convolution

MobileNet uses depthwise separable convolution to reduce the number of parameters:

1. Depthwise convolution: Applies a spatial filter independently to each input channel.
2. Pointwise convolution: Uses a (1 \times 1) convolution to combine information across channels.

For standard convolution with kernel size (K \times K), (C_{\text{in}}) input channels, and (C_{\text{out}}) output channels, the parameter count is:

[ K^2 C_{\text{in}} C_{\text{out}} ]

For depthwise separable convolution, it is:

[ K^2 C_{\text{in}} + C_{\text{in}}C_{\text{out}} ]

Assuming equal input and output channel counts (C), the reduction factor is:

[ \frac{K^2C^2}{K^2C+C^2} ]

With (K=3) and (C=32), this is approximately a sevenfold reduction.

Supervised Image Classification

A classifier’s raw scores, or logits, are commonly converted into class probabilities using softmax, which produces values between 0 and 1 that sum to 1.

Cross-entropy loss compares predicted probabilities with the true labels. For one-hot labels, it reduces to the negative log-probability assigned to the correct class.

Using Models When Target Data Is Limited

Training a new model from scratch may be impractical when there is not enough labeled data. The lecture outlines three ways to reuse a model trained on another task or domain:

1. Feature extraction: Use features from a chosen layer without tuning the model.
2. Fine-tuning: Train or adjust selected network layers using target-task data.
3. Unsupervised domain adaptation: Adapt the model using unlabeled data from the target domain.

The lecture notes that these transfer approaches will be developed further in the next lecture.

Speakers and Sources

• Speaker: One lecturer, whose name is not stated in the subtitles.
• Source: NPTEL IIT Delhi lecture video.