Passing the blame backwards

The network

Inputs x₁ = 1.0 and x₂ = 0.5, target y = 0.1. The hidden neurons use the sigmoid σ from puff 6, and the loss L is the squared miss:

h₁ = σ(w₁x₁ + w₂x₂)
h₂ = σ(w₃x₁ + w₄x₂)
ŷ = w₅h₁ + w₆h₂    L = (ŷ − y)²

The chain rule

A small change in w₁ changes h₁’s total, which changes h₁, which changes ŷ, which changes L. The chain rule says the overall slope is the product of the slopes of each link:

∂L∂w₁ = 2(ŷ − y) · w₅ · h₁(1 − h₁) · x₁

Here h(1 − h) is the slope of the sigmoid. Backpropagation computes the shared front part once, as a “delta” for each neuron, starting at the end:

δ_out = 2(ŷ − y) = ?
δ₁ = δ_out · w₅ · h₁(1 − h₁) = ?
δ₂ = δ_out · w₆ · h₂(1 − h₂) = ?

Then every weight’s blame is the delta at its far end times the value at its near end: ∂L/∂w₅ = δ_outh₁, ∂L/∂w₁ = δ₁x₁, ∂L/∂w₂ = δ₁x₂, and so on. The update is w ← w − η ∂L/∂w with learning rate η = 0.1.

Check it by nudging

You could get the same slope the slow way: nudge one weight a tiny bit, run forward again, and see how much the loss moved. On this page, nudging w₁ up by 0.0001 gives a slope of ?; backpropagation says ?.

Why this matters

Nudging needs one extra forward pass per weight. With six weights that’s fine; with a billion it means a billion forward passes for a single step. Backpropagation gets every slope from one forward pass and one backward pass, and the backward pass costs about the same as the forward one, because it does the same multiplications in reverse and reuses each delta for every weight that feeds into it.

What this page leaves out

Real neurons also have a bias (puff 6); it gets blamed the same way, with blame equal to its neuron’s delta. Real training averages the blame over a batch of many examples instead of one. And deep-learning libraries do all of this automatically, for any network you write, which is called automatic differentiation. The idea was popularised for neural networks by Rumelhart, Hinton and Williams in 1986, building on earlier work on computing slopes backwards (Linnainmaa, 1970; Werbos, 1974).