Watch a small network learn

The network

Each dot’s position (x, y), with both between −1 and 1, goes into a hidden layer of n neurons. Each one computes a weighted sum plus a bias and squashes it with tanh:

h_i = tanh(a_ix + b_iy + c_i)
tanh(z) = e^z − e^−ze^z + e^−z

tanh is an S-curve like the sigmoid of puff 6, but it runs from −1 to 1 and is centred on zero, which tends to make learning smoother. With the second layer on, another n tanh neurons take the first layer’s outputs as their inputs. Finally one output neuron adds everything up and squashes it with the sigmoid:

p = σ(v₁h₁ + … + v_nh_n + d)

p is the network’s probability that the dot is blue. The shading shows p for every spot.

The loss and the nudges

The loss is the cross-entropy from puff 6, −log of the probability given to the right colour, averaged over the dots. Each step uses mini-batch gradient descent: pick 20 dots at random, work out every knob’s slope by backpropagation (puff 8), average the slopes over the 20, and move each knob by η × slope. A small random batch makes each step cheap and a little noisy, which usually doesn’t matter. Right now: not started.

Counting the knobs

Each hidden neuron has 2 weights and a bias, and the output neuron has one weight per hidden neuron plus its own bias: 4n + 1 knobs. A second layer adds n × n weights and n biases. This network has 9.

The same recipe, with many refinements, trains chatbots. GPT-3, for example, had 175 billion knobs.

What this page simplifies

The knobs start at small random values (each weight scaled by 1/√inputs, biases at zero), so two runs can end differently: press Reset to roll new ones. Real training uses cleverer update rules than a plain nudge, far more data, and checks the network on dots it never trained on (puff 5).