LECTURE 18

Summary

The lecture continues the discussion of generative adversarial networks (GANs), focusing on conditional generation and image-to-image translation.

Conditional GANs

• A basic GAN generates outputs without being given a desired class or other condition.
• A conditional GAN (cGAN) adds condition information—often a class label—to the generator’s input.
• The lecture illustrates this with MNIST: early in training, generated outputs are unclear; with more training, the generator produces recognizable digits organized by class.
• The lecture also mentions a related setup in which the discriminator performs two tasks: distinguishing real from generated samples and predicting their class.

Pix2Pix: Paired Image-to-Image Translation

Pix2Pix translates an input image into an output image using paired examples during training. Its loss combines:

• Adversarial loss, which encourages realistic outputs.
• L1 loss, which encourages the generated image to match its paired target.

Dropout is applied in several generator layers to introduce stochasticity. In this setup, dropout is used during both training and testing.

The discriminator evaluates the input and output together, learning to distinguish real image pairs from pairs containing a generated image.

Potential applications mentioned include:

• Converting OpenStreetMap images into realistic aerial imagery.
• Increasing image resolution, such as translating lower-resolution satellite imagery into higher-resolution imagery.
• Translating synthetic aperture radar (SAR) images into optical images, including when optical observations are unavailable.

A key limitation of Pix2Pix is its need for paired training data.

CycleGAN: Unpaired Image-to-Image Translation

CycleGAN addresses the paired-data requirement by learning translations between two domains using unpaired images. Example domains include summer and winter scenes, SAR and optical imagery, or low- and high-resolution imagery.

Its framework contains:

• Two generators: one translating from domain A to B, and another from B to A.
• Two discriminators: one for each domain, distinguishing real images from generated images.
• Two translation cycles:
  • A → generated B → reconstructed A.
  • B → generated A → reconstructed B.

Loss Components

1. Adversarial losses: One for each domain’s generator–discriminator pair, encouraging generated images to look like they belong to the target domain.
2. Cycle-consistency loss: Typically an L1 loss between an original image and the image recovered after translating it to the other domain and back. This helps preserve content and structure and discourages arbitrary outputs.
3. Identity loss: Typically an L1 loss that encourages a generator to leave an image unchanged when it is already in the target domain.

The lecture ends by noting that the discussion will continue in the next lecture.

Speakers and Sources

• Speaker: Unnamed lecturer, presenting the NPTEL lecture.
• Source: NPTEL, IIT Delhi.