Finite differences

01A finite difference estimates a derivative by evaluating the function at nearby inputs#

A finite difference estimates a derivative by evaluating the function at nearby inputs. It can check a derivative implementation because it uses function values rather than the same symbolic derivative rules. The result is an approximation with a step-size tradeoff.

A centered finite difference samples the function on both sides of the input. With a positive step $h$, the endpoints are $x-h$ and $x+h$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Estimate $f\prime(x)$ by dividing the output difference by the full spacing $2h$:

For $h=0.1$, values $f(x-h)=1.7$ and $f(x+h)=2.3$ give an estimate $(2.3-1.7)/0.2=3$.

For $f(x)=x^2$ at $x=2$ with $h=0.1$, the centered numerator is $f(2.1)-f(1.9)=4.41-3.61=0.8$. Divide by $2h=0.2$ to get 4. The denominator is the full distance between the two sampled inputs, not the distance from the center to one sample.

02A finite step gives an estimate, which need not equal the exact derivative#

A finite step gives an estimate, which need not equal the exact derivative. For $f(x)=x^3$ at $x=1$ with $h=0.5$, the endpoint outputs are $0.125$ and $3.375$. The estimate is $3.25$; the exact derivative there is $3$.

Use both endpoint evaluations even when the value at the center is known.

03Both endpoints must lie in the domain#

Both endpoints must lie in the domain. A centered slope across a corner need not be a derivative. For smooth functions, subtracting nearly equal floating-point outputs can lose accuracy.

Smaller steps can reduce approximation error while increasing roundoff. They do not always improve the estimate.

04A closer look#

A large step may average over curvature; an extremely small one can magnify numerical rounding when nearly equal outputs are subtracted. Finite-difference agreement is useful evidence for a gradient implementation, but one successful point does not check every input or edge case.