CalculusConcept reference

Reverse-mode differentiation

Reverse-mode differentiation propagates output sensitivities backward through local derivative rules.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Understand the idea
If a downstream scalar L=L(y) supplies ∂ L/∂ y=4, start the backward pass with 4
A backward pass reuses the forward values to evaluate local derivative rules
A downstream loss can supply an initial sensitivity other than 1
Key takeaway
Sources & further reading
Concept connections

01Understand the idea#

Reverse-mode differentiation computes how a final scalar responds to earlier values by working backward through a computational graph. Neural-network backpropagation is an application of this method. The backward numbers are sensitivities, not the original forward values run in reverse.

First evaluate the graph forward. Then initialize the final scalar’s sensitivity to itself as 1. At each preceding node, multiply the received sensitivity by the appropriate local derivative, evaluated using the saved forward inputs.

For a scalar output $y$ , write $\bar u=\partial y/\partial u$ for its sensitivity to a node $u$ . Start at the output: $\bar y=\partial y/\partial y=1$ .

Move backward through the graph, multiplying the received sensitivity by the operation’s local derivative.

Let $u=3x$ and $y=u^2$ . At $x=2$ , first compute $u=6$ and $y=36$ . The square’s local rate is $2u=12$ ; the scale’s local rate is $3$ .

Figure 1A schematic backward pass for u=3x and y=u² at x=2. Forward values are x=2,u=6,y=36. Sensitivity starts at dy/dy=1, goes to dy/du=12, then dy/dx=36. Backward edges multiply by 12 and 3 respectively.

For $u=3x$ , $y=u^2$ at $x=2$ , save $u=6$ and $y=36$ . Backward from $y$ , begin with 1. The square contributes $2u=12$ , so sensitivity to $u$ is 12. Multiplying by $du/dx=3$ gives sensitivity to $x$ equal to 36. The two occurrences of 36 here have different meanings: output value and input sensitivity.

Check your reasoning

$u=2x$ and $y=u^2$ . At $x=5$ , find the backward sensitivity $dy/dx$ .

Show answer and explanation

The forward value is $u=10$ . Backward multiplication gives $1\times(20)\times(2)=40$ .

02If a downstream scalar L=L(y) supplies ∂ L/∂ y=4, start the backward pass with 4#

If a downstream scalar $L=L(y)$ supplies $\partial L/\partial y=4$ , start the backward pass with $4$ . Multiply by each local rate at the forward evaluation point.

\frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\frac{dy}{du}\frac{du}{dx}

Check your reasoning

$u=x^2$ and $y=3u$ . At $x=1$ , a downstream scalar $L=L(y)$ supplies $\partial L/\partial y=-2$ . Find $\partial L/\partial x$ .

Show answer and explanation

-12

The local rates are $dy/du=3$ and $du/dx=2$ . Multiply backward: $-2\times(3)\times(2)=-12$ .

03A backward pass reuses the forward values to evaluate local derivative rules#

A backward pass reuses the forward values to evaluate local derivative rules. It does not perturb each input. For $u=3x$ , $y=u^2$ , keeping only $2u$ gives the response to $u$ , not to $x$ .

Continue to the requested input and include every local factor along that chain.

Check your reasoning

$u=-4x$ and $y=u^2$ . At $x=1$ , a backward pass stops at $dy/du$ . Repair the final sensitivity $dy/dx$ .

Show answer and explanation

The forward value is $u=-4$ . Backward multiplication gives $1\times(-8)\times(-4)=32$ .

04A downstream loss can supply an initial sensitivity other than 1#

A downstream loss can supply an initial sensitivity other than 1. If $\partial L/\partial y=4$ for this same example, sensitivities become 48 at $u$ and 144 at $x$ . Use the supplied upstream sensitivity; resetting it to 1 would differentiate $y$ instead of the requested $L$ .

Key takeaway

Start with the output sensitivity and multiply through every operation back to the requested input.

Propagate a scalar-output sensitivity backward through a chain.

Sources & further reading

[1]
Stanford CS231n: Backpropagation ↗Stanford CS231n · Article

01Understand the idea#

02If a downstream scalar L=L(y) supplies ∂ L/∂ y=4, start the backward pass with 4#

03A backward pass reuses the forward values to evaluate local derivative rules#

04A downstream loss can supply an initial sensitivity other than 1#

Start with the output sensitivity and multiply through every operation back to the requested input.

Sources & further reading

Reference this concept