LECTURE 15

Summary

The lecture continues the discussion of pretext-task-based self-supervised learning and considers how tasks first developed in computer vision can be adapted to Earth observation (EO) data. A pretext task lets a model learn from the data itself—such as by predicting spatial or temporal relationships—without manually supplied labels.

Pretext tasks from computer vision

• Context prediction

  • Divide an image into a grid, such as a 3 × 3 arrangement.
  • Choose the central patch and one of its eight neighboring patches.
  • Train a network to classify the neighbor’s position relative to the central patch.
  • This encourages the model to learn object structure and scene layout.
  • Its usefulness for EO depends partly on spatial resolution. At very high resolution, relationships such as a building beside a road may be clear. At coarser resolutions, local spatial relationships may be less prominent because each pixel covers a larger area.

• Rotation prediction

  • Rotate an image and train a model to identify the rotation.
  • This is generally less suitable for top-view EO imagery than for ordinary front-view photographs: a rotated aerial image may not look obviously rotated, making the task difficult to infer reliably.

• Image jigsaw puzzle

  • Divide an image into patches, shuffle them, and train the model to identify the permutation or recover the original arrangement.
  • Instead of using every possible permutation—which could create too many classes—a selected set of permutations can be assigned class labels.
  • Patch representations can be processed with weight-shared CNNs.
  • As with context prediction, the task’s usefulness in EO depends on spatial resolution and may be greater for high-resolution imagery.

Choosing task difficulty

Self-supervised learning can benefit from more challenging pretext tasks, but only up to a point. If a task is too easy, it may provide a weak learning signal; if it is effectively impossible, performance and learning effectiveness can decline.

One way to make a jigsaw task harder is to remove some input information—for example, by converting selected patches from multiple channels to grayscale. The goal is to increase difficulty without making the correct answer impossible to infer.

Tasks particularly relevant to Earth observation

• Geolocation classification: When latitude–longitude metadata is available, group images into geographic clusters and train the model to predict which cluster an image belongs to.

• Band prediction and colorization: Give the model some image channels and ask it to predict a missing channel. The lecture relates this to image colorization, where the L channel in LAB color space is used to predict the A and B channels. In EO, a similar approach is to drop a band and predict it from the remaining bands, or to predict a derived quantity such as NDVI. The lecture also notes that related approaches have been used with synthetic aperture radar imagery.

• Image inpainting: Mask a salient region and train the model to restore it. Spatial resolution is a consideration when judging whether this task is suitable for aerial imagery. Inpainting is also an important idea in self-supervised learning with vision transformers, which will be covered later in the course.

Extending ordering tasks to EO data

• Time-series imagery: Like video frames, acquisitions in an image time series have a temporal order. A pretext task can shuffle the acquisitions and ask the model to recover their order or identify the shuffle. Unlike ordinary video, EO acquisitions may be separated by enough time for substantial real-world changes to occur. The consequences depend on the application—for example, earthquake-related urban change may be rapid, while some agricultural changes may be more gradual.

• Hyperspectral imagery: A single hyperspectral image contains many spectral bands with an ordered relationship; neighboring bands are generally related. A model can be given bands in shuffled order and asked to detect the shuffle, restore the order, or identify the permutation that was applied.

Overall, the lecture emphasizes that pretext tasks should be selected with the properties of the EO data in mind—especially spatial resolution, temporal spacing, and spectral structure—rather than transferred from generic computer vision without adjustment.

Speakers and sources

• Speaker: One lecturer or instructor, not identified by name in the subtitles.
• Source/channel: NPTEL IIT Delhi.