The Algorithm That Taught Machines to Learn: How Back-Propagation (1986) Became the Engine of Modern AI

Every time you ask ChatGPT a question, talk to Claude, or let Gemini summarise your inbox, you are watching a 40-year-old idea do its quiet work. The models behind them were trained by an algorithm called back-propagation — and the paper that made it famous appeared in Nature on 9 October 1986, written by David Rumelhart, Geoffrey Hinton and Ronald Williams under a modest title: “Learning representations by back-propagating errors.”

It is one of the most consequential papers in the history of computing. This is a tribute to what it proposed, the thirty-year problem it cracked, the honest story of who really discovered it, and how a single rule for learning from mistakes ended up training almost every AI you have ever touched.

1. The wall neural networks hit

The dream of a machine that learns like a brain is old. In 1958 Frank Rosenblatt built the perceptron, a simple network that could be trained to tell patterns apart. But a single layer of these artificial neurons could only separate categories with a straight line — and in 1969 Marvin Minsky and Seymour Papert proved, in their book Perceptrons, that such a network could not learn even the humble XOR function (true when exactly one of two inputs is on). The verdict was so deflating that funding drained away and the field slid into its first “winter.”

Everyone knew the fix in principle: stack more layers, with hidden units in between, and a network could carve up the world in far richer ways. But that raised a brutal question no one could answer efficiently — the credit-assignment problem. When a deep network gets an answer wrong, which of its thousands of interior weights deserves the blame, and by how much? Without a way to apportion that blame, the hidden layers simply could not be trained.

2. The idea: send the mistake backwards

Back-propagation is the answer to that question, and at heart it is astonishingly simple: let the network learn from its own errors, working backwards.

Training happens in two sweeps. In the forward pass, an example flows through the network and produces an output, which is compared with the desired answer to give an error. Then, in the backward pass, that error is propagated back through the network layer by layer, using the chain rule of calculus to compute exactly how much each individual weight contributed to the mistake. Each weight is then nudged a little in the direction that reduces the error — a step of gradient descent.

Here E is the error, w is any weight in the network, and η (the learning rate) sets the size of each step. The power is in the term ∂E/∂w — the gradient — which back-propagation computes for every weight in a single efficient backward sweep, rather than testing each one by hand. Repeat over many examples and the whole network gradually descends towards weights that get things right.

One quiet requirement made it all work: the neurons had to be smooth. The perceptron’s hard on/off threshold has no usable slope to follow, so the authors used gently curved activation functions (such as the sigmoid) whose gradients the error could flow through.

3. The payoff: networks that invent their own concepts

The paper’s title is the real headline. Trained this way, a network’s hidden units learn to represent things nobody programmed into them. In the abstract’s own words:

In one celebrated demonstration, the authors trained a network on a set of family relationships. With no instruction beyond examples, its hidden units spontaneously discovered abstract features such as a person’s generation, nationality, and which side of the family they belonged to. The network had, in effect, invented useful concepts on its own — the first clear glimpse of what we now call learned representations, the beating heart of deep learning.

4. Who really invented back-propagation?

Here is the honest history, and it makes the story richer rather than smaller. Rumelhart, Hinton and Williams did not discover back-propagation first — they made it land. The underlying mathematics, known more generally as reverse-mode automatic differentiation, had been found and re-found several times over.

What the 1986 paper had that the others lacked was a clear, compelling demonstration and the right moment. It convinced a generation of researchers that multi-layer networks really were trainable and worth pursuing — and that conviction, more than priority, is why the field eventually took off.

5. From one paper to every model you use

Back-propagation turned out to be a universal training engine. Change the architecture, and the same backward sweep of error still teaches it:

Every one of these is trained, at bottom, by computing an error and sending it backwards. The 2017 Transformer — the subject of an earlier tribute in this series — is only trainable because back-propagation scales gracefully to hundreds of billions of weights.

6. A prize-winning idea

The world eventually caught up with the significance of this line of work. In 2018 the ACM — computing’s leading professional society — gave its A.M. Turing Award, often called the Nobel Prize of computing, to Yoshua Bengio, Geoffrey Hinton and Yann LeCun “for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.”

Then, in a striking sign that these ideas had reshaped science itself, the 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” A method for teaching machines to learn had become part of the story of physics.

Why it still matters

Almost every headline about AI today — a new model, a new capability, a new record — rests on the quiet arithmetic this paper popularised. Its genius was not a single trick but a reframing: that a network does not need to be told what its interior should represent if it can instead be corrected, gently and repeatedly, by its own mistakes. That simple shift made hidden layers trainable, trainable hidden layers made depth useful, and depth made everything that followed possible.

Some papers describe the world. A rare few teach the machines that are now helping to build it. Learning representations by back-propagating errors is one of them.

Sources & further reading

• D. E. Rumelhart, G. E. Hinton, R. J. Williams, ‘Learning representations by back-propagating errors’, Nature 323, 533–536 (1986), DOI 10.1038/323533a0
• Wikipedia: Backpropagation · Perceptrons (Minsky & Papert, 1969)
• J. Schmidhuber, ‘Who Invented Backpropagation?’ (on Linnainmaa 1970, Werbos 1974 and the earlier roots)
• 2018 ACM A.M. Turing Award (Bengio, Hinton, LeCun) · 2024 Nobel Prize in Physics (Hopfield, Hinton)
• Image: multi-layer neural-network diagram by Offnfopt, via Wikimedia Commons, released into the public domain (CC0)

Curated by Jerry Cards - jerrycards.com. Our 致敬 (tribute) series celebrates the landmark papers and discoveries that quietly built the modern world. More at jerrycards.com/news.