Frobenius norm

01Turn a matrix of errors into one size#

A model can make several predictions for each example in a batch. Subtract the targets and you get a matrix of errors. To compare two predictions, we often need one number that measures the size of that whole matrix.

Simply adding the entries would let positive and negative errors cancel. A residual of $-4$ is still an error of size four. The Frobenius norm counts every entry by squaring it before adding.

Start with a familiar vector. The length of $(3,-4)$ is $\sqrt{3^2+(-4)^2}=5$. The Frobenius norm uses the same operation on all the coordinates of a matrix.

You can imagine laying the entries out in one long vector. Their arrangement into rows and columns does not change this length. We are measuring entry sizes, rather than how the matrix transforms a vector.

02Compute the norm entry by entry#

Consider this matrix. There are four entries to include, including the negative one.

The first row contributes $1^2+(-2)^2=5$. The second contributes $2^2+4^2=20$. Add those contributions, then take the square root: $\sqrt{25}=5$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For a matrix with $m$ rows and $n$ columns, the definition is

Here $A_{ij}$ means the entry in row $i$, column $j$. The two sums visit every column in every row. The double bars mean a norm, and the subscript $F$ specifies the Frobenius norm. The result is one nonnegative number.

03Keep the norm and its square distinct#

Our example has two useful quantities: $\|A\|_F=5$ and $\|A\|_F^2=25$. The squared norm is just the sum of squared entries. Taking a square root when a formula asks for the squared norm changes the quantity.

If the entries measure errors in meters, the norm has units of meters; the squared norm has units of square meters. The outer exponent in $\|A\|_F^2$ matters.

For entries $3$ and $-4$, adding absolute values gives $7$, while the square-root-of-squares calculation gives $5$. These are different ways to measure size.

Also include entries away from the diagonal. The Frobenius norm is not a determinant, a trace, or the square of a matrix product. Its definition is an entry-by-entry calculation.

Nothing in the definition requires a square matrix. For the rectangular matrix below, the zero entries contribute zero, and all six positions are included.

The squared entries total $1+4+0+4+0+16=25$, so $\|B\|_F=5$. Different matrices can have the same norm; the single size does not identify the individual entries.

04Measure the difference between two matrices#

To measure the distance between two equally shaped matrices, first subtract corresponding entries. For example,

Subtracting corresponding entries gives

The distance is $\|A-B\|_F=\sqrt{3^2+4^2}=5$. Entries where the matrices agree contribute nothing.

The order of operations matters: form the residual matrix, then measure it. In general, $\|A-B\|_F$ is not $\|A\|_F-\|B\|_F$.

For instance, matrices containing the single entries $3$ and $-3$ each have norm $3$. Their norms differ by zero, but their entries differ by $6$. Measuring each matrix separately loses the information needed for distance.

05Read a matrix loss#

In a prediction problem, write $R=\widehat Y-Y$ for the residual matrix. A commonly chosen loss is

Read this in stages: square every residual, add the squares, then divide by $2m$. Here $m$ is the number of examples. The norm notation itself does not perform any averaging.

For two examples with two outputs each, suppose all four residuals equal $1$. The squared Frobenius norm is $4$, and this loss is $4/(2\cdot2)=1$.

This choice averages the summed output errors over examples, with an extra factor of one half. Averaging over every entry would use a different divisor. Always read the normalization around the norm, as well as the norm itself.

This size treats entries equally in their current units. If one output is measured in thousands and another in fractions, their squared contributions may be very different. Choosing units or weights is part of designing a loss; the Frobenius norm tells you exactly what the chosen formula counts.

Further questions