Latent Space Visualisation: PCA, t-SNE, UMAP | Deep Learning Animated

Summary

The video explains three methods for reducing high-dimensional data—often an autoencoder’s latent space—to two or three dimensions for visualization: PCA, t-SNE, and UMAP. The methods differ in the structures they preserve, how they work, and how fast or easy they are to tune.

PCA: Principal Component Analysis

PCA projects data onto directions that capture the greatest variation.

The video illustrates this with measurements such as height and weight. Data may vary more along one direction than another; these directions are called principal components. To reduce dimensions, PCA keeps the components with the largest associated eigenvalues and projects the data onto them.

The process is:

1. Center the data by subtracting each variable’s mean.
2. Compute the covariance matrix from the centered data.
3. Find the covariance matrix’s eigenvectors and eigenvalues.
4. Sort the eigenvectors by descending eigenvalue. The leading eigenvectors are the principal components.
5. Project the data onto the selected components.

Choosing a principal component can preserve important structure better than projecting onto an arbitrary axis. In the video’s example, PCA retains three patient groups that simple axis projections merge.

Strengths: PCA is fast, relies on linear algebra, and is relatively interpretable. For latent-space visualizations, its eigenvalues also indicate how much variation each component captures.

Limitation: PCA uses linear projections, so it can struggle with nonlinear relationships—for example, spiral-shaped data whose groups cannot be separated by a straight-line projection.

SNE and t-SNE: Preserving neighborhoods

The video introduces Stochastic Neighbor Embedding (SNE) as the basis for t-distributed Stochastic Neighbor Embedding (t-SNE). The central idea is to keep points that are neighbors in the original high-dimensional space close together in the lower-dimensional representation.

The SNE process is described as follows:

1. For each point, convert its distances to other points into a Gaussian-based probability distribution. The probabilities represent how likely other points are to be considered its neighbors.
2. Adjust the Gaussian’s width, or variance, to control how many points count as neighbors.
3. Repeat this for the points in the high-dimensional space, then create a corresponding probability distribution for the low-dimensional points.
4. Compare the two distributions using Kullback–Leibler (KL) divergence.
5. Minimize that divergence by adjusting the low-dimensional positions.

The video describes perplexity as the main SNE/t-SNE tuning parameter. It is used to select the Gaussian widths and is related to the effective number of neighbors. In practice, the right value may need to be found by testing different settings.

Strength: t-SNE can represent nonlinear structure that PCA may fail to separate, such as distinct groups in a spring-like dataset.

Limitations: SNE is computationally slow. t-SNE modifies SNE to improve its performance and visualizations, including by using symmetric distributions in the high-dimensional space and a Student’s t-distribution in the low-dimensional space. Even so, the video presents t-SNE as slower than PCA and UMAP.

Perplexity can substantially change the visualization, sometimes even after a small adjustment, making results time-consuming to tune. The video also notes that larger perplexity can increase runtime.

UMAP: Graph-based dimensionality reduction

UMAP also aims to preserve neighborhood relationships, but represents them with weighted graphs rather than the probability distributions used by SNE and t-SNE.

The video outlines these steps:

1. Find each sample’s nearest neighbors in the high-dimensional data.
2. Connect each point to its selected neighbors. The number of neighbors is a key parameter.
3. Convert those connections into weighted edges. The video describes applying exponential decay to distances, offset by the distance to the closest neighbor.
4. Combine the per-point graphs into one graph by symmetrizing edges between pairs of points.
5. Build a corresponding weighted graph for the low-dimensional representation.
6. Compare the graphs using their adjacency matrices and a cross-entropy loss.
7. Minimize the loss with stochastic gradient descent, adjusting the low-dimensional positions.

Strengths: The video presents UMAP as fast, effective for nonlinear data, and comparatively predictable to tune. Its example visualizations show well-separated MNIST classes and better preservation of some global structure.

Parameter behavior: Increasing the number of nearest neighbors gradually changes the amount of detail in the projection. The video characterizes this behavior as easier to interpret than the sometimes dramatic changes caused by adjusting t-SNE’s perplexity.

Overall guidance

• Start with PCA when you want a quick, simple visualization.
• Consider t-SNE when nonlinear local group structure matters, while keeping in mind its runtime and sensitivity to perplexity.
• Consider UMAP as a general choice. The narrator prefers it among the three based on the video’s comparisons of speed, visualization quality, and parameter behavior.
• The video also mentions TriMAP and PaCMAP as more recent dimensionality-reduction methods.

Speakers and sources featured

• Unnamed narrator — explains the methods and comparisons.
• Laurens van der Maaten and Geoffrey Hinton — cited as the developers of t-SNE.
• Geoffrey Hinton — cited in connection with the earlier SNE method and mentioned alongside other deep-learning contributions.
• Yoshua Bengio and Yann LeCun — mentioned alongside Hinton as prominent figures in deep learning.