Forward evaluation

01A forward pass evaluates a computation from known inputs to outputs#

A forward pass evaluates a computation from known inputs to outputs. At every operation, substitute the values produced by its input nodes. Storing these intermediate results makes a complicated calculation traceable and supplies the values needed to evaluate derivatives later.

A forward pass computes the graph’s values. Start with known inputs; evaluate a node only after every value it uses is available.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Here $x=2$, $u=x+4$, and $y=u^2$. Compute $u=6$ first. The square then receives $6$, giving $y=36$.

Squaring the original $x$ would give $4$, which evaluates a different expression.

For $a=x+1$, $b=a^2$, and $y=b-2a$ at $x=2$, calculate $a=3$ before $b=9$. Then compute $y=9-2(3)=3$. The shared value $a=3$ is used twice. Replacing the second use with the original input 2 would change the function.

02Branches may be evaluated in either order if they do not depend on each other#

Branches may be evaluated in either order if they do not depend on each other. With $x=2$, $u=x^2$, $v=3x$, and $y=u+v$, compute $u=4$ and $v=6$ before the join.

03The dependency arrows decide what comes next#

The dependency arrows decide what comes next. A node needs its immediate inputs, even if the expression contains the same original variable several times.

Keep each intermediate value until all of its consumers have used it.

04A closer look#

Before debugging a gradient, check that the forward pass computes the intended prediction and loss. A perfectly implemented derivative of the wrong computation still trains the wrong model. Each stored value belongs to the current input and parameter setting.