LECTURE 17

Summary

The lecture moves from contrastive self-supervised learning to generative adversarial networks (GANs), introducing their components, objective, training challenges, and a conditional version.

From contrastive learning to GANs

Contrastive self-supervised methods such as SimCLR use positive and negative sample pairs. Their loss functions often benefit from large batches because the denominator includes many negative examples.

Large batch requirements are a drawback. The lecture raises two related research challenges:

• How can a method maintain many negative samples without using huge batches?
• Can contrastive learning work without negative samples altogether?

Later methods addressed these challenges in part, for example by using memory banks to retain negatives or architectural changes that remove the need for negative samples. The lecture defers their details to future discussion.

How a GAN works

A GAN consists of two competing models:

• Generator (G): Takes a noise or latent vector (z) as input and produces synthetic data (G(z)).
• Discriminator (D): Receives real samples (x) and generated samples (G(z)), and tries to distinguish between them.

The discriminator aims to classify real and synthetic samples correctly, while the generator aims to fool the discriminator. In the standard objective, the discriminator maximizes the objective and the generator minimizes it.

The lecture motivates this competition with two examples:

• Counterfeiters trying to make fake currency convincing while law enforcement tries to detect it.
• A cuckoo bird’s eggs mimicking a host bird’s eggs while the host tries to identify and reject them.

Ideally, if both models have sufficient capacity and training succeeds, the generator’s distribution approaches the real data distribution. The discriminator then cannot reliably tell them apart, and its output is approximately one-half for either source.

GAN training challenges and approaches

Training may fail to produce a meaningful competition—for instance, if the discriminator becomes so strong that the generator cannot improve effectively. The lecture describes several ways to help balance training:

• Adjust update frequency or learning rates: Update the generator and discriminator at different rates to compensate if one is stronger.
• Use a buffer of previously generated images: Mix older generated samples with current ones when training the discriminator, helping prevent it from overfitting to the generator’s latest outputs.
• Train progressively across resolutions: Start with low-resolution images and gradually move to higher resolutions, allowing the model to learn increasingly complex patterns. The lecture notes this can be useful in earth-observation settings.

Example: Synthesizing hyperspectral data

For a basic GAN that synthesizes hyperspectral data:

1. The generator takes a latent noise vector and produces a synthetic spectrum.
2. The discriminator receives both real spectra and generated synthetic spectra.
3. It learns to distinguish between real and synthetic data, while the generator learns to make its spectra more convincing.

Conditional GANs

A standard GAN’s generator is conditioned only on its latent vector. A conditional GAN (cGAN) adds meaningful information—often a class label—as another input.

• The condition can be a class such as building, road, or vegetation.
• The label is represented as an embedding and combined with the latent vector.
• The generator uses both the noise and the condition to produce data associated with that condition.

The lecture introduces conditional GANs for further discussion in the next lecture.

Speakers and sources featured

• Speaker: An unnamed lecturer; no individual name is given in the subtitles.
• Sources or examples mentioned: SimCLR; the GAN paper’s counterfeit-currency example; the cuckoo-bird and host-bird analogy; hyperspectral data; and earth-observation applications.