The idea: neurons hold numbers, layers pass them on
This page follows the classic first example of a neural network: reading a handwritten digit. The image is 28 × 28 = 784 pixels, and each pixel is one number between 0 and 1. Those 784 numbers are the first layer. Two hidden layers of 16 neurons each follow, and the last layer has 10 neurons, one for each digit 0 to 9. Every neuron holds one number, its activation. The brightest output neuron is the network's answer.
Why layers? Recognizing a 9 can be split into smaller questions: is there a loop at the top, is there a line on the right? A loop in turn is made of small edges. The hope is that each layer answers questions built from the answers of the layer before. Whether a trained network really does that is a different matter (see What it really learned).
Neurons and activations
An input neuron's activation is just its pixel's brightness: 0 for white paper, 1 for full ink (MNIST images are stored the other way round, white ink on black; only the colours differ). On the canvas a darker circle means an activation closer to 1. The 10 output activations say how much the image looks like each digit. For the clean 3 the output "3" is 0.999 and every other output is below 0.003. (Demo 1: forward pass)
Weights are a picture, the bias is a threshold
A neuron in hidden layer 1 is connected to all 784 pixels, with one weight on each connection. It adds up weight × pixel over all 784 pixels and then adds its bias:
z = w1·a1 + w2·a2 + … + w784·a784 + bDraw the 784 weights in the pixels' places and you get a 28 × 28 picture: blue where the weight is positive, red where it is negative. Ink on a blue pixel raises the sum, ink on a red pixel lowers it, and blank pixels contribute nothing. A neuron that should detect one edge would have a blue stripe with red on both sides. The bias says how high the weighted sum must be before the neuron turns on: with b = −10 it stays near 0 until the sum passes 10.
Neuron 1 of hidden layer 1 on the 3: the products on blue pixels add up to +8.25, those on red pixels to −5.21, and the bias is +1.80, so z = 4.84 and a = σ(4.84) = 0.99. (Demo 2: one layer-1 neuron) Neurons in hidden layer 2 and in the output layer work the same way, but with 16 weights, one per neuron of the layer before. (Demo 3, Demo 4)
Sigmoid squishification
The weighted sum z can be any number, but an activation should be between 0 and 1. The sigmoid (logistic) function does the squishing: σ(z) = 1 / (1 + e−z). Very negative z gives almost 0, very positive z almost 1, and σ(0) = 0.5. This network uses the sigmoid in all three layers, as in the lesson it is based on.
Modern networks mostly use ReLU instead: ReLU(z) = max(0, z). It is cheaper and trains more easily in deep networks, because the sigmoid is almost flat for large |z| and passes back tiny gradients. The sigmoid came from the picture of a biological neuron that is either firing or not; ReLU just turned out to work better.
A layer in one line: σ(W·a + b)
Put the 784 activations in a column vector a(0), and the weights of the 16 hidden neurons as the 16 rows of a matrix W (16 × 784). Then all 16 weighted sums are one matrix-vector product, and the whole layer is
a(1) = σ(W(1)·a(0) + b(1)) (16×784)·(784×1) + (16×1) = (16×1)
a(2) = σ(W(2)·a(1) + b(2)) (16×16)·(16×1) + (16×1) = (16×1)
a(3) = σ(W(3)·a(2) + b(3)) (10×16)·(16×1) + (10×1) = (10×1)σ is applied to each entry separately. Libraries and GPUs are built to compute exactly these products fast. The small 3-input network on Neural Network: the Forward Pass shows the same product row by row.
13,002 weights and biases
| layer | weights | biases | total |
|---|---|---|---|
| hidden 1 | 784 × 16 = 12,544 | 16 | 12,560 |
| hidden 2 | 16 × 16 = 256 | 16 | 272 |
| output | 16 × 10 = 160 | 10 | 170 |
| network | 12,960 | 42 | 13,002 |
Almost all of them sit in the first layer, because every pixel connects to every first-layer neuron. (Demo 5: count the parameters)
The network is just a function
Put together, the network is one big function: 784 numbers in, 10 numbers out, with 13,002 knobs that decide what it computes. Nothing in it "knows" what a digit is. Learning means finding values for the 13,002 knobs that make the right output light up on example images; that is gradient descent with gradients from backpropagation.
What it really learned
The weights on this page were really trained, on 20,000 synthetic hand-drawn digits (three stroke styles per digit, randomly rotated, scaled, sheared and moved by a pixel or two). On new digits of the same kind it is right every time. But open its first-layer neurons and they do not look like neat edges: they are blurry blobs of blue and red. The network found some way to separate the training digits, not the tidy edges-then-loops plan.
It also fails in ways a person never would:
| input | network says | why |
|---|---|---|
| the 3, shifted 7 pixels right | 2 (0.83) ✗ | it only saw digits near the centre; the weights are tied to pixel positions |
| the 7, drawn small | 1 (1.00) ✗ | it only saw digits of about one size |
| a blank image | 1 (0.97) ✗ | every input gets an answer; with no ink, only the biases speak |
| random noise | 0 (0.33) | no output above 0.5: at least here it is unsure |
(Demos 6–8) Convolutional networks fix the shifting problem by sliding the same small weight pattern over every position (see Convolution in a CNN); more varied training data fixes the size problem; a network that always has to pick one of 10 digits has no way to say "this is not a digit".
A network's first layer is a set of learned pictures; for the version with fixed pictures, the average image of each digit, and why it reads only 82 % of MNIST, see Draw a Digit: Template Matching.
What the page leaves out
How the weights were trained (cost function, gradient descent, backpropagation) is not animated here; see Backpropagation: What One Example Wants (this network trained from random weights), Backpropagation and Gradient Descent. The training digits are synthetic strokes, not the real MNIST set, so the accuracy says little about real handwriting (networks of this size reach about 96–97 % on MNIST). You cannot draw your own digit here; the 14 inputs are fixed (to draw one, see Draw a Digit: Neural Network Math). The weights are rounded for the page (layer 1 to 8 bits), and the page computes with the rounded values.