The idea: rates multiply

Three gears mesh: x drives u, and u drives y. Gear x has 24 teeth, u has 12 and y has 8. When x turns once, its 24 teeth push 24 teeth of u, so u turns 24 / 12 = 2 times. Each turn of u pushes 12 teeth of y, so y turns 12 / 8 = 1.5 times per turn of u. Per turn of x, y turns 2 × 1.5 = 3 times. The rates multiply. That is the chain rule:

dy/dx = dy/du · du/dx

Three meshing gears in a row with 24, 12 and 8 teeth, labelled x, u and y. Below them: u turns 2 times per turn of x, y turns 1.5 times per turn of u, so y turns 3 times per turn of x.
Gear ratios multiply: 2 × 1.5 = 3 turns of y for every turn of x.

Here u = 2x and y = 1.5u, so y = 3x. Forward turns x by a quarter turn: u turns 0.5 and y turns 0.75. (Demo: gears, rates multiply) Meshing gears also turn in opposite directions; the page tracks only how far each one turns.

A nudge flows through the chain

A derivative says how much the output moves for a small move of the input. Move x by a small Δx. Then u moves by about Δu ≈ (du/dx)·Δx, and y moves by about Δy ≈ (dy/du)·Δu. Put them together: Δy ≈ (dy/du)·(du/dx)·Δx. Divide by Δx and let it shrink: that is the chain rule again.

For y = sin(x²) at x = 1 with Δx = 0.01 (Demo: nudge x by 0.01):

nodeestimateexact change
xΔx = 0.010.01
u = x²2 × 0.01 = 0.021.01² − 1 = 0.0201
y = sin u0.5403 × 0.0201 = 0.01086sin 1.0201 − sin 1 = 0.01069

Δy/Δx = 1.069, close to the exact dy/dx = 1.0806. The estimate is not exact because the functions curve; with Δx = 0.001 the ratio is 1.079, and it reaches 1.0806 in the limit (see the derivative as a limit). For the gears everything is a straight line, so the nudge gives exactly 3.

Forward and backward on a computation graph

A program computes a formula in small steps. Write each step as a node: u = x², then y = sin u. This is a computation graph. Each edge has a local derivative: how the node changes with its input, on its own. du/dx = 2x and dy/du = cos u.

  • Forward pass: compute the values left to right. x = 1, u = 1, y = sin 1 = 0.8415.
  • Backward pass: start at the output with dy/dy = 1. Walk right to left. At each edge, multiply what arrived by the local derivative: dy/du = 1 × cos 1 = 0.5403, then dy/dx = 0.5403 × 2 = 1.0806.
The graph x to u = x squared to y = sin u. Above it, blue values flow left to right: x = 1, u = 1, y = 0.8415. Below it, red gradients flow right to left: dy/dy = 1, dy/du = 0.5403, dy/dx = 1.0806, with the local derivatives 2x = 2 and cos u = 0.5403 on the edges.
Values flow forward; gradients flow backward, multiplied by one local derivative per edge.

The local derivatives use the values from the forward pass (cos u needs u), so the forward pass always comes first. (Demo: sin(x²) forward, backward, numeric check)

Longer chains: one factor per link

A longer chain only adds factors. For y = esin(x²) the graph is x → a = x² → b = sin a → y = eb:

dy/dx = dy/db · db/da · da/dx = eb · cos a · 2x = 2.3198 × 0.5403 × 2 = 2.5068 at x = 1.

The backward pass computes the product from the right, one link at a time, and keeps each partial product (dy/db, dy/da) on the way. (Demo: deeper chain)

Branches: the gradients of the paths add

When x feeds two nodes, a nudge of x reaches y along two paths, and the two effects add. So the backward pass adds the gradients that arrive at a node from its children. For y = u · v with u = x² and v = sin x, the local derivatives of the product are ∂y/∂u = v and ∂y/∂v = u:

dy/dx = ∂y/∂u · du/dx + ∂y/∂v · dv/dx = v · 2x + u · cos x

That is the product rule, (uv)′ = u′v + uv′, read as a sum over two paths. At x = 1: 0.8415 × 2 + 1 × 0.5403 = 1.683 + 0.5403 = 2.2232. (Demo: branching x² · sin x)

A diamond graph. x at the left feeds u = x squared at the top and v = sin x at the bottom, both feed y = u times v at the right. The top path multiplies v times 2x = 1.683, the bottom path multiplies u times cos x = 0.5403, and dy/dx = 1.683 + 0.5403 = 2.2232.
Multiply along each path, then add the paths: the product rule in picture form.
graph shaperulebackward pass does
chain x → u → ydy/dx = dy/du · du/dxmultiply along the edge
x feeds two nodesdy/dx = sum over pathsadd what arrives from each child
y = u · vproduct rulelocal derivatives v and u

Checking with a numeric slope

Check slope evaluates the whole function twice and takes the central difference (f(x+h) − f(x−h)) / 2h with h = 0.0001. It needs no derivatives at all, so it is a good independent check. For sin(x²) it gives 1.080605, the backward pass 1.080605. Machine-learning libraries do this as a gradient check when they test a new layer. It is too slow for training: one pair of evaluations per input, against one backward pass for all inputs at once.

Where it is used

  • Backpropagation is this backward pass on a neural network: thousands of nodes, one seed dy/dy = 1 at the loss, local derivatives multiplied and paths added.
  • Gradient descent uses the gradient from the backward pass to step downhill.
  • Taylor series and Newton's method need derivatives of composed functions, found with the chain rule.

Read backwards, the chain rule becomes u-substitution: ∫f(g(x)) g′(x) dx = ∫f(u) du, drawn as stretched strips on integration by parts and substitution.

For the product rule as a growing rectangle u·v (strips v·du and u·dv), see Derivative Rules as Pictures.

What the page leaves out

Only one input x is shown; with many inputs the same backward pass gives all partial derivatives at once (the gradient). The forward-mode alternative (pushing derivatives left to right, like the nudge) is shown only as the nudge. The page does not cover the quotient rule, implicit differentiation, the multivariable chain rule with Jacobian matrices, or how a library such as PyTorch records the graph while the program runs.