Backpropagation (Theory)
The mathematical foundation for computing gradients in neural networks.
In Plain Language
A network of little parts guesses an answer and gets it wrong. Now you need to know which parts to blame. Backpropagation works backwards from the mistake, handing each part its share of the blame, so every part knows which way to adjust. Without it you would have to test each part separately, which would take far too long.
Training a network means adjusting thousands of weights, and to adjust one you need to know how much it contributed to the error. Backpropagation computes all of those contributions in a single backward sweep: start with the error at the output, then apply the chain rule from calculus layer by layer, so each layer's share of the blame is derived from the layer after it. The alternative — nudging each weight and re-running the whole network to see what changed — costs one full pass per weight, which is hopeless at any real scale. Linnainmaa formulated the method in 1970 and Werbos applied it to networks in 1974, but it only became consequential once there was hardware that could run it.
Why It Was Important
Though popularized in the 1980s, the application of the chain rule to continuously differentiable multi-stage networks was first formulated by Seppo Linnainmaa (1970) and Paul Werbos (1974). This algorithm efficiently calculates the error derivative for network weights, which is the absolute backbone of all modern deep learning.
Who Invented It
Seppo Linnainmaa / Paul Werbos
Mathematicians who derived reverse-mode automatic differentiation.
Applications
- Deep Learning Training
- Gradient Descent
- Optimization
Videos
Neural Networks Pt. 2: Backpropagation Main Ideas
StatQuest with Josh Starmer