Shared-path gradients

01A value can influence the final loss through more than one branch#

A value can influence the final loss through more than one branch. Its backward sensitivity must add every branch’s contribution. This is the multivariable chain rule applied to a computational graph: multiply along each route, then add where the routes meet.

A shared input can affect an output along several paths. Multiply rates along each path, then add the contributions where they meet.

For $u=x^2$, $v=3x$, and $y=u+v$, both branches depend on $x$.

At $x=2$, the square branch contributes $2x=4$ and the scale branch contributes $3$. The sum passes sensitivity $1$ into each branch.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Let $a=x^2$, $b=3x$, and $L=a+b$. At $x=2$, the square route contributes $1\times2x=4$ and the linear route contributes $1\times3=3$. The total derivative is 7. Overwriting the first contribution with the second would incorrectly leave only 3.

02For u=2x and L=u^2+3u at x=1, the intermediate value is u=2#

For $u=2x$ and $L=u^2+3u$ at $x=1$, the intermediate value is $u=2$. Add its two branch rates: $4+3=7$. Then apply the upstream rate $du/dx=2$.

03Contributions carry signs#

Contributions carry signs. For $y=x^2-4x$ at $x=2$, they are $4$ and $-4$, giving a total rate of $0$. Two nonzero paths can cancel.

A backward implementation adds into a shared node’s sensitivity. Replacing its old value loses earlier paths.

04Shared weights and reused activations are common in models#

Shared weights and reused activations are common in models. Their gradients accumulate contributions from each use. A node’s number of outgoing edges alone is not the gradient; each route has its own derivative, evaluated at the saved forward values.