LECTURE 30

Summary

The lecture covers deep-learning approaches to hyperspectral image unmixing and enhancement, focusing on denoising, super-resolution, and inpainting.

1. Deep-Learning Approaches to Unmixing

• Linear mixing model (LMM): A pixel’s spectrum is modeled as a weighted combination of endmember spectra. The weights represent the fractional abundances of materials in the pixel.
• Abundances are commonly constrained to be nonnegative and sum to one, making the mixture interpretable.
• CNN-based unmixing:
  • A 1D CNN can take a pixel’s spectral vector as input and learn hierarchical spectral features. A 3D CNN is another possible extension.
  • Output neurons correspond to endmembers. A sigmoid or softmax can produce values between zero and one, interpreted as abundances.
  • One described training strategy uses a binary target vector indicating which materials are present, rather than training directly on known abundance fractions. The lecturer notes that the network’s outputs are nevertheless treated as abundance estimates.
• Endmember extraction: Pure-material spectra may not be known in advance, so they sometimes need to be estimated alongside abundances. Methods that assume pure pixels exist may fail in scenes where all pixels are mixtures.
• Autoencoder-based unmixing:
  • The encoder compresses a pixel spectrum into a low-dimensional latent vector, interpreted as its abundance vector.
  • The decoder reconstructs the original spectrum using a simple linear transformation.
  • For example, with 200 bands and five endmembers, the bottleneck has five dimensions and the decoder maps back to 200 dimensions.
  • The decoder’s weight matrix has one column per endmember. During training, the latent representation is encouraged to encode abundances, while the decoder weights are encouraged to represent endmember spectra.
  • Constraints such as nonnegativity may be needed to make the latent values valid abundances.

2. GAN Background

The lecture briefly reviews generative adversarial networks (GANs):

• A generator maps a latent vector to a synthetic sample.
• A discriminator attempts to distinguish real samples from generated ones.
• The generator learns to make its outputs difficult to distinguish from real data.
• GANs can be applied to hyperspectral data synthesis and enhancement.
• In a conditional GAN, additional information—such as a class label or an input image—conditions generation. Image-to-image translation methods such as pix2pix and CycleGAN are mentioned as related examples.

3. Hyperspectral Image Enhancement

Enhancement is presented as a broad term that includes denoising, super-resolution, and inpainting. It is useful because hyperspectral images can be noisy, limited in spatial resolution, or affected by missing or corrupted regions.

A key challenge is to preserve both spatial detail and spectral relationships across hundreds of bands. Noise can also vary from band to band.

Denoising

• Conventional filtering:
  • Spatial filters, such as mean and Gaussian filters, process neighboring pixels but can blur detail and have limited ability to model spectral information.
  • Spectral filtering can help preserve continuity across bands and may be combined with spatial methods.
• Self-supervised denoising:
  • Start with clean images and add simulated noise.
  • Train a model to reconstruct the original clean image from the corrupted input.
  • A denoising autoencoder is one example of this encoder–decoder setup.
• GAN-based denoising:
  • A generator reconstructs hyperspectral patches from noisy patches.
  • Reconstruction loss encourages recovery of the input’s content, while adversarial loss encourages realistic outputs.
• Noise is not necessarily uniform across bands. Some bands may be more corrupted than others, so methods may identify and treat low-noise and high-noise bands separately. The lecture also notes that some especially problematic bands may be dropped during preprocessing.
• Gaussian noise is commonly used in synthetic experiments, but real imagery can contain more complex patterns, including stripe-like or spike noise.

Super-Resolution

Super-resolution aims to create higher-spatial-resolution hyperspectral imagery, enabling finer-grained analysis.

• Single-input super-resolution:
  • Artificially downsample a higher-resolution training image.
  • Use the degraded image as the model input and the original image as the target.
  • Train the model to map the lower-resolution input to the higher-resolution target.
  • At inference, the approach assumes the learned mapping can enhance an available image to an even higher resolution.
• GAN-based super-resolution:
  • A generator creates high-resolution hyperspectral images from lower-resolution inputs.
  • A discriminator encourages realistic spatial detail.
  • Reconstruction losses can help preserve the accuracy of spectral signatures.
• Representative-band super-resolution:
  • Apply super-resolution to selected spectral bands, chosen using band-selection methods.
  • Use high-resolution information from those bands to guide reconstruction of other bands, or enhance only the selected bands if that suits the application.
  • This can reduce computational cost.
• Fusion or pan-sharpening-style approaches:
  • Combine a low-resolution hyperspectral image, rich in spectral information, with a higher-resolution RGB, multispectral, or single-band image, rich in spatial detail.
  • Fuse the two inputs to improve reconstruction.

Inpainting

Inpainting restores missing or corrupted image regions while aiming to preserve spatial and spectral consistency.

• Traditional interpolation estimates missing pixels from nearby values but becomes limited when missing regions are large.
• Self-supervised inpainting:
  • Intentionally mask pixels, patches, or regions during training.
  • Give the partially observed image to the model and train it to reconstruct the masked areas.
  • Because the targets are generated by masking the image itself, external labels are not required.
• GAN-based inpainting:
  • A generator predicts missing regions using surrounding spatial and spectral information.
  • A discriminator evaluates whether the reconstructed regions appear realistic.

4. Sensor Shift and Open Problem

Models trained for one hyperspectral sensor may not transfer effectively to another because the sensors can have different band combinations. The lecturer leaves this as a homework question: consider how to benefit from deep networks’ learned features when moving to a new sensor, potentially without retraining the network.

Speakers and Sources

• Speaker: One lecturer or instructor, unnamed in the subtitles.
• Other speakers or sources: None identified.