Local derivatives

01A local derivative describes one operation’s response to one of its immediate inputs#

A local derivative describes one operation’s response to one of its immediate inputs. It temporarily ignores where that input came from. This lets a large model reuse small derivative rules for operations such as addition, multiplication, and squaring.

A local derivative measures one operation’s response to one of its input ports. Hold its other ports fixed. The notation $\partial z/\partial u$ asks how this gate’s output $z$ responds to input $u$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For $z=uv$, hold $v$ fixed. A change $\Delta u$ changes the output by $v\Delta u$, so the local rate is $v$.

At $u=3$, $v=5$, the output is $15$ but the local rate on $u$ is $5$.

For one multiplication node $z=uv$ at $u=3,v=-2$, changing $u$ slightly scales that change by $v$, so $\partial z/\partial u=-2$. Changing $v$ scales it by $u$, giving $\partial z/\partial v=3$. The forward value $z=-6$ is neither of these derivatives.

02Use the derivative rule for the gate itself#

Use the derivative rule for the gate itself. For $z=u^2$, the local rate is $2u$. For $z=u+v$, with $v$ fixed, it is $1$. For $z=ku$, with fixed $k$, it is $k$.

The rate may depend on the gate’s current input value, so use the value from the forward pass.

03If u=3x and z=u^2, the gate’s local rate is ∂ z/∂ u=2u#

If $u=3x$ and $z=u^2$, the gate’s local rate is $\partial z/\partial u=2u$. The rate with respect to the original $x$ also needs $du/dx=3$.

Those are different requested derivatives. Include the upstream factor only when the question asks for the response to that upstream input.

04A local derivative is one factor in a longer chain#

A local derivative is one factor in a longer chain. If $u$ itself depends on a weight, you must still include that dependence when finding the loss’s response to the weight. Stopping at a local derivative answers a different question.