One neuron

The neuron

Call the temperature x₁ (in °C) and the chance of rain x₂ (in %). The neuron multiplies each by its weight, adds the bias b, and squashes the total z with the sigmoid function σ:

z = w₁x₁ + w₂x₂ + b
p = σ(z) = 11 + e^−z

Right now the neuron computes z = 0.2 x₁ + 0 x₂ − 4.

Why the border is straight

The answer is exactly 0.5 when z = 0, because σ(0) = 1/(1 + 1) = 0.5. So the border is every day where w₁x₁ + w₂x₂ + b = 0, and that is the equation of a straight line. Here it is temperature = 20 °C. With three inputs the border would be a flat plane, and with more it is the same idea in more dimensions. It is never curved.

How it learns

Write y = 1 for days we walked and y = 0 for days we stayed in. A good score for a yes/no answer is the cross-entropy, the average of −log of the probability the neuron gave to what really happened:

L = −1n Σ [ y log p + (1 − y) log(1 − p) ]

Saying 0.9 on a day we walked costs −log 0.9 ≈ 0.11. Saying 0.1 costs −log 0.1 ≈ 2.3, so confident mistakes are punished hard. On this page the loss is now 0.5. Its slopes come out neatly:

∂L∂w_j = 1n Σ (p − y) x_j
∂L∂b = 1n Σ (p − y)

“Let it learn” nudges every knob against its slope, exactly as in puff 1. One neuron trained this way is called logistic regression. As in puff 1, the page rescales the two inputs to similar sizes while learning (learning rate 1 in those units) and shows the weights in °C and %. Because a straight line can split these sixteen days perfectly, the loss could keep shrinking for ever as the weights grow, so the page stops once every day is right and the loss is below 0.1.

Where the idea came from

In 1943 Warren McCulloch and Walter Pitts described a neuron as a unit that adds up its inputs and fires if the total passes a threshold. In 1958 Frank Rosenblatt’s perceptron learned its weights from examples. It used a hard on/off step instead of the smooth sigmoid; the smooth version is what lets us compute slopes. In 1969 Marvin Minsky and Seymour Papert’s book Perceptrons stressed the catch you just saw: a single unit can only separate things with a straight line.

How much like a brain is it?

Only loosely. A real nerve cell has thousands of connections, sends spikes over time and runs on chemistry that this sum ignores. The artificial neuron borrows one idea, “add up the signals and react if there’s enough”, and nothing more.