Why layers can bend the line

The activation

An activation function is what a neuron does to its total after adding up. ReLU (rectified linear unit) keeps positive totals and zeroes the rest:

ReLU(z) = max(0, z)

The network on this page

One input x, a hidden layer of n ReLU neurons (hidden because you only see their effect through the output), and one output neuron that adds them up:

f(x) = v₁ ReLU(w₁x + b₁) + …
+ v_n ReLU(w_nx + b_n) + c

To keep the picture honest and simple, the page fixes each w_i = 1 and sets b_i so the hinges sit evenly across the range, at x = 5. The output knobs v_i and c are then found exactly by least squares, the same “best fit” formula as in puff 1, just with more knobs: v = …. The typical miss is …. Because the hinges can’t move, one or two of them can even do slightly worse than a plain straight line: half of a lone hinge is stuck flat. In a real network the hinge positions are learned too, by the nudging you’ll see in the next two puffs, so they move to wherever bends are needed.

Why it collapses without the bend

Without an activation, a layer just multiplies by a table of weights W and adds biases b. Do it twice and you can multiply out the brackets:

W₂(W₁x + b₁) + b₂
= (W₂W₁)x + (W₂b₁ + b₂)

That is one layer again, with weights W₂W₁. However many layers you stack, it all folds into a single straight line. With the bends removed, the best this page’s network can do is y = …. Many settings of the knobs give that same line, so the figure shows the one closest to the knobs the network had with its bends, which lets you see each neuron’s own straight line.

How far can bending go?

The universal approximation theorem says one hidden layer with enough neurons can match any smooth curve as closely as you like, on a limited range. George Cybenko proved it in 1989 for S-shaped activations, Kurt Hornik in 1991 for a much wider family, and Leshno and colleagues in 1993 showed that any activation except a plain polynomial works, ReLU included. The catch: it promises that good weights exist, not how many neurons you need or how to find them. In practice networks stack several layers, because layers that bend the bends of the layer before can make far more pieces from the same number of neurons (Montúfar and colleagues, 2014). ReLU itself became popular after Nair and Hinton used it in 2010.