Truncated SVD

01A large matrix can contain patterns that repeat across many entries#

A large matrix can contain patterns that repeat across many entries. Truncated SVD stores an approximation using only the strongest patterns. Each retained component has one pattern across rows, one across columns, and a strength. Their outer product fills a whole matrix.

For an image, entries can be brightness values. A few components may reproduce broad structure while omitting detail. For other matrices, entries might be measurements rather than pixels. This lesson reconstructs a matrix from supplied components; computing those components is a separate numerical task.

An SVD writes a matrix as weighted rank-one pieces. A rank-$k$ truncation keeps the $k$ largest singular scales and their matched vector pairs. Each retained piece contributes a whole matrix.

For leading scale 5, $u=(0.6,0.8)$ and $v=(1,0)$, compute every product $5u_iv_j$. The first column is $(3,4)$ and the second is zero.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Build a rank-one term in two steps. For $u=(0.6,0.8)$ and $v=(1,0)$, the outer product has first row $(0.6\times1,0.6\times0)=(0.6,0)$ and second row $(0.8,0)$. Multiplying every entry by the singular value 5 gives rows $(3,0)$ and $(4,0)$. Do not multiply only the diagonal.

02Keep large components, not small ones#

Keep large components, not small ones. If scales are 6 and 2, the rank-1 truncation keeps $6u_1v_1^T$. Keeping the other term is a different matrix, even though it also has rank 1.

03To keep two components, add both weighted matrices#

To keep two components, add both weighted matrices. If unit outer products have rows $C_1:(1,0);(0,0)$ and $C_2:(0,0);(0,1)$, scales $(5,2)$ give rows $(5,0);(0,2)$. Keeping every nonzero component recovers the original matrix.

For two components, make two weighted matrices before adding. If $C_1$ has rows $(1,0);(0,0)$ and $C_2$ has rows $(0,0);(0,1)$, then $5C_1$ and $2C_2$ have rows $(5,0);(0,0)$ and $(0,0);(0,2)$. Entrywise addition gives $(5,0);(0,2)$. Keep each scale paired with its own two vectors.

04Explore the visual intuition#

Truncating to the largest singular values gives a best approximation of rank at most $k$ in Frobenius norm. A small matrix error is useful for compression, but does not guarantee that the lost detail is unimportant for a particular prediction task. The next lesson measures exactly how much squared matrix error was discarded.

Further questions