Loading timeline…
20122010s
Dropout
Randomly switching off neurons during training so a network cannot rely on any one of them.
Why It Was Important
Dropout was the regularization technique that made large networks trainable without immediately overfitting, and it was one of the practical ingredients behind AlexNet's 2012 ImageNet result. By forcing units to work in changing company it approximates training an ensemble of networks at once, and it remains a standard component of deep learning architectures.
Who Invented It
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov
Hinton's group at the University of Toronto.
Applications
- Image Classification
- Regularization
- Speech Recognition
- Language Models
Key Papers
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava et al. · Journal of Machine Learning Research · 2014