Principal components

01Understand the idea#

Principal component analysis, or PCA, finds new perpendicular feature directions that capture as much variation in centered observations as possible, one direction at a time. It is useful when many measured features move together and you want a smaller set of coordinates.

Imagine a cloud of centered two-feature observations stretched along a diagonal. The first principal direction follows its longest spread. The second is perpendicular and describes the remaining spread. A direction is a unit vector shared by all observations; each observation will get its own coordinate along it.

Put observations in the rows of $X_c$ and features in columns. In $X_c=U\Sigma V^T$, the right singular vectors live in feature space, so they are the PCA directions. Larger singular values correspond to more variation along their paired directions. The factors are supplied in these tasks.

Put centered observations in the rows of $X_c$. In its SVD, principal directions are the columns of $V$. They live in feature space and are ordered by descending singular values.

With scales $(5,2)$ and $V$ rows $(0.6,-0.8);(0.8,0.6)$, the first direction is column 1: $(0.6,0.8)$. The second is $(-0.8,0.6)$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Read the matrix name before choosing entries. If $V$ has rows $(0.6,-0.8)$ and $(0.8,0.6)$, its first column is $(0.6,0.8)$ and its second is $(-0.8,0.6)$. Both have squared length 1. Multiplying the first by its scale 5 gives a stretched vector, not a unit principal direction.

02A closer look#

The singular scale describes spread along a direction; it does not multiply the unit direction. With distinct scales, reversing a direction’s sign changes its orientation but not its line. These tasks keep the supplied signs.

03When the factor is supplied as V^T, use its rows#

When the factor is supplied as $V^T$, use its rows. If $V^T$ has rows $(0.8,0.6);(-0.6,0.8)$, those rows are the two principal directions in order. Do not take its columns again.

Transposition swaps rows and columns. If the supplied object is $V^T$, each principal direction is already a row. Label it before extracting a vector. Direction 2 means the vector paired with the second singular value; it does not mean the second coordinate of direction 1.

04PCA can turn correlated sensor readings into fewer coordinates for plotting or modeling#

PCA can turn correlated sensor readings into fewer coordinates for plotting or modeling. It uses feature variation without target labels. A low-variance direction might still predict the target; a high-variance one may mostly reflect measurement units. Evaluate the reduced representation for its intended use.

Further questions