Explained variance

01After choosing PCA directions, you need to decide how many to keep#

After choosing PCA directions, you need to decide how many to keep. Explained variance reports the share of the centered dataset’s variation represented by selected components. It is a property of this dataset and transformation, not the fraction of future predictions that will be correct.

Singular values measure lengths. Variance measures squared spread, so each component contributes its squared singular value. For sample covariance there is also a common divisor $n-1$; it cancels in a fraction because retained and total variance use the same observations.

Retained variance compares squared singular scales. Square the complete list, add the retained squares, and divide by the sum of all squares. The total must be positive.

For singular values $(4,3)$, total squared scale is $16+9=25$. Keeping the first component retains $16/25$ of the variance, not $4/7$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

With complete singular values $(4,3)$, square first to obtain $(16,9)$. Total variation is proportional to $25$; keeping the first component retains $16/25=0.64$, or 64%. The discarded share is $9/25=0.36$. Using $4/(4+3)$ would compare lengths instead of variation.

02For scales (5,4,3), keeping two retains (25+16)/50=41/50#

For scales $(5,4,3)$, keeping two retains $(25+16)/50=41/50$. The discarded fraction is $9/50$. These add to 1 because they partition the same squared total.

03A retained fraction describes variation in the centered data#

A retained fraction describes variation in the centered data. It does not guarantee predictive quality. Compare it to a target only after squaring: a target of at least $4/5$ includes equality.

For scales $(5,4,3)$, total squared scale is $25+16+9=50$. One component retains $25/50=0.5$; two retain $41/50=0.82$. A target of at least 80% therefore needs two components in this example. Compare the cumulative fraction to the target after computing it.

04A variance target gives a reproducible compression rule#

A variance target gives a reproducible compression rule. Before using it for prediction, validate the model with the chosen transformation fitted only on training data. A component carrying little feature variation may still carry useful label information.

Further questions