Loading timeline…
20142010s
Attention Mechanism
A mechanism allowing models to automatically focus on relevant parts of the input sequentially.
Why It Was Important
Introduced by Dzmitry Bahdanau for sequence-to-sequence translation, attention solved the bottleneck of compressing long sentences into a fixed-length vector. By allowing the decoder to 'look back' at specific words in the source sentence when predicting the next word, it radically improved machine translation.
Who Invented It
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
The MILA research group in Montreal.
Applications
- Machine Translation
- Pre-cursor to Transformers
- Speech Recognition
Key Papers
- Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · ICLR 2015 · 2014
Videos
What are Transformers (Machine Learning Model)?
IBM Technology
Attention for Neural Networks, Clearly Explained!!!
StatQuest with Josh Starmer