Unbiased gradients

01Average over the sampling scheme#

A sampled gradient is unbiased for a target objective when its expectation, at fixed parameters, equals that objective's full gradient. The sampling probabilities are part of the claim.

For a mean loss over $n$ examples, the full gradient is the average of their gradients. Sampling one example uniformly gives that same average in expectation, because each example has probability $1/n$.

02Compare two sampling rules#

At fixed parameters, suppose two scalar example gradients are $-2$ and $-8$. The gradient of their mean objective is $(-2-8)/2=-5$. Uniformly choosing either raw example gradient has expectation $-5$.

If the first example is chosen with probability $0.9$ and the second with $0.1$, the expected raw gradient becomes $0.9(-2)+0.1(-8)=-2.6$. This sampling rule is biased for the original equally weighted objective.

03State which objective is targeted#

The nonuniform sampler is unbiased for a different weighted objective whose weights are $0.9,0.1$. Calling an estimator biased or unbiased requires naming the target, not merely examining whether the sample looks random.

For vector gradients, compare each coordinate's expectation. Unbiasedness does not make an individual draw equal to the full gradient, and it does not guarantee that every stochastic step decreases loss.

Sampling convenient or easy examples can change the expected gradient. Importance weighting can correct some nonuniform schemes, but deriving such corrections is outside this lesson. Here, compute the expected supplied gradient under the stated probabilities and compare it with the intended target.