Lecture 14: Singular Value Decomposition (Part-2) | Singular Values as Eigen Values | GO Classes DA

Summary

The lecture develops the Singular Value Decomposition (SVD), explains its relationship to eigenvalues and matrix rank, and previews how it can reveal a matrix’s fundamental subspaces.

1. SVD works for any matrix

For any real (m \times n) matrix (A), the SVD has the form

[ A = U\Sigma V^{T}, ]

where (U) and (V) are orthogonal matrices, and (\Sigma) is a rectangular diagonal matrix containing the nonnegative singular values. Unlike ordinary eigenvalue decomposition, SVD does not require (A) to be square or symmetric.

The instructor contrasts SVD with eigenvalue decomposition, which applies to square matrices and, when the matrix is diagonalizable, can be written as

[ A = X\Lambda X^{-1}. ]

For symmetric matrices, the eigenvectors can be chosen to be orthogonal.

2. Finding an SVD

The procedure outlined in the lecture is:

1. Compute (A^{T}A) or, alternatively, (AA^{T}).
2. Find the eigenvalues and eigenvectors of (A^{T}A). The eigenvectors form the right singular vectors, which are the columns of (V).

3. Take the square roots of the eigenvalues. If the eigenvalues are (\lambda_i), the corresponding singular values are [ \sigma_i = \sqrt{\lambda_i}. ] The lecture recommends listing the singular values in descending order.

4. Find the left singular vectors for positive singular values. For each (\sigma_i > 0), [ u_i = \frac{Av_i}{\sigma_i}. ] Alternatively, the columns of (U) can be found from the eigenvectors of (AA^{T}).

5. Complete (U) if needed. If the matrix dimensions require more columns than the positive singular values provide, extend the existing orthonormal vectors to an orthonormal basis. The lecture briefly mentions basis extension and Gram–Schmidt as ways to do this.

3. Singular values as an eigenvalue-like relation

For each positive singular value (\sigma_i), the SVD gives

[ Av_i = \sigma_i u_i. ]

This resembles the eigenvector equation (Ax = \lambda x), but the input vector (v_i) and output vector (u_i) can differ. This formulation also works when (A) is rectangular. In matrix form, these relations are expressed as

[ AV = U\Sigma. ]

4. Rank and singular values

A key result is

[ \operatorname{rank}(A) = \text{the number of nonzero singular values of } A. ]

The lecture motivates this result through (A^{T}A):

• (A^{T}A) is symmetric, and its eigenvalues are the squares of (A)’s singular values.

• The null spaces of (A) and (A^{T}A) are equal:

  • If (Ax = 0), then (A^{T}Ax = 0).
  • If (A^{T}Ax = 0), then [ x^{T}A^{T}Ax = |Ax|^{2} = 0, ] so (Ax = 0).

• Equal nullities, together with the rank–nullity theorem, imply [ \operatorname{rank}(A) = \operatorname{rank}(A^{T}A). ]

• Because (A^{T}A) is symmetric, its rank equals the number of its nonzero eigenvalues. These correspond exactly to the nonzero singular values of (A).

The instructor also emphasizes that for a general matrix, the rank is not necessarily the number of nonzero eigenvalues. For a symmetric matrix, the rank does equal the number of nonzero eigenvalues, counted with multiplicity. This is why applying the eigenvalue argument to the symmetric matrix (A^{T}A) is useful.

5. SVD and the four fundamental subspaces

The lecture previews how the singular vectors reveal the fundamental subspaces:

• The left singular vectors corresponding to positive singular values form an orthonormal basis for the column space of (A).
• The right singular vectors corresponding to zero singular values form an orthonormal basis for the null space of (A).
• The row space and left null space can also be obtained from the SVD, but their detailed treatment is left for a later class.

The lecture closes by noting that SVD connects to many linear algebra topics and is useful in machine learning. Further discussion of geometry, basis extension, and practice questions is deferred to later lessons.

Speakers and sources featured

• Instructor: Unnamed lecturer leading the class.
• Live-class participants or commenters: Deepak Kakashi (the name may be garbled in the captions), Tjas, Anjali Kulkarni, and Aditya/Adity (spelling varies in the captions).
• Sources or materials mentioned: GATE Overflow, a University of California, Berkeley source, a textbook attributed in the captions to David C. Lay, and earlier course videos or playlists on eigenvalues, rank, and fundamental subspaces.