Gradient checks

01Understand the idea#

A gradient check compares a program’s derivative with a finite-difference estimate of the same function at the same point. It helps catch missing factors, signs, and coordinate mixups before using the gradient to train a model.

The check must specify the discrepancy metric and tolerance. Absolute error and relative error answer different questions, especially near zero. Compute the stated metric instead of deciding that two displayed numbers merely look close.

A gradient check compares an analytic derivative with a numerical estimate at the same input. An explicit tolerance decides whether the discrepancy is acceptable.

For absolute tolerance $\tau$, pass when $|a-n|\le\tau$. Equality passes. Treat displayed decimals as exact in these exercises.

With analytic derivative $a=2$, numerical estimate $n=2.03$, and $\tau=0.05$, the difference is $0.03\le0.05$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

If the analytic derivative is 2.01 and the finite-difference estimate is 2.00, their absolute difference is 0.01. Under the relative metric $|a-n|/(|a|+|n|)$, it is $0.01/4.01\approx0.00249$. A tolerance of 0.001 fails this comparison. Always use the denominator and zero-case rule specified by the task.

02A mixed tolerance uses an absolute allowance t and a relative allowance r#

A mixed tolerance uses an absolute allowance $t$ and a relative allowance $r$. Let $m=\max(|a|,|n|)$, the larger absolute derivative value. These values set the allowed discrepancy:

For $a=10$, $n=10.1$, $t=0.01$, and $r=0.02$, the larger magnitude is $10.1$. The allowance is $0.01+0.02(10.1)=0.212$.

The discrepancy is $0.1$. Because $0.1\le0.212$, this point passes the mixed rule.

03A mismatch needs investigation#

A mismatch needs investigation. Check the analytic rules, the numerical step and precision, and whether both evaluations used the same fixed inputs. A corner can also invalidate the comparison.

A passing point does not test all inputs or coordinates. A numerical estimate is not an exact reference by default. Recheck fresh points and coordinates instead of treating one pass as a proof.

04Check several inputs and coordinates, including cases with shared branches#

Check several inputs and coordinates, including cases with shared branches. A passing sample does not prove the whole gradient correct. Near a nondifferentiable point, finite differences can mix behavior from opposite sides, so disagreement needs interpretation rather than an automatic formula change.