Computational graphs

01Understand the idea#

A computational graph breaks a formula into small operations connected by their dependencies. It shows which intermediate values must exist before later values can be computed. Training systems use the same dependency structure to propagate derivatives backward.

A computational graph names the small operations inside an expression. Each arrow points from an input value to an operation that uses it.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For $y=(x+3)^2$, first name the sum $u=x+3$. Then write $y=u^2$. The square receives $u$, so the edge runs from $u$ to $y$.

These node equations also describe the graph in text: every variable on the right is an incoming dependency.

For $y=(x+2)^2$, create a node $u=x+2$ and then a node $y=u^2$. At $x=3$, the first node gives 5 and the next gives 25. The edge from $u$ to the square means the square consumes $u$. An edge is a dependency, not an instruction to add the two node labels.

02Operation order changes the expression#

Operation order changes the expression. Squaring first and then adding $3$ gives $x^2+3$. To represent $(x+3)^2$, the add node must come first.

Each node can be one small operation. A whole model layer is not required.

03A value may feed more than one operation#

A value may feed more than one operation. For $y=x(x+1)$, write $u=x+1$ and $y=xu$. The input $x$ feeds both the addition and the final multiplication.

Substitute the intermediate node into the output node to check that the graph represents the intended formula.

04If an intermediate result is reused, draw its outgoing edges from the same node#

If an intermediate result is reused, draw its outgoing edges from the same node. That shared node matters later: its effect on a loss includes every downstream use. Replacing a dependency graph with the written order of symbols can hide those shared paths.