Transformer Architecture
The defining neural network architecture, built on 'self-attention' instead of recurrence.
In Plain Language
Older programs read a sentence one word at a time, like reading through a straw, and had often forgotten the beginning by the time they reached the end. A transformer looks at every word at once and lets each word check which other words matter to it. In "she poured it into the cup", the word "it" can glance back and find what was poured.
Earlier sequence models processed text one token at a time, carrying a running summary forward — which made long-range connections fragile and forced the computation to happen in order. The transformer drops that entirely. Every token compares itself against every other token and produces weights for how much each of them matters to it; that is self-attention. Because those comparisons do not depend on one another, they can all run at once, which is what let these models exploit GPUs and grow to sizes nobody had trained before. The architecture is the reason "scale up the data and the parameters" became a viable strategy at all, and essentially every large language model since is a variation on it.
Why It Was Important
The paper 'Attention Is All You Need' dispensed with recurrence (RNNs/LSTMs) entirely, relying strictly on self-attention to draw global dependencies between input and output. Because it processes all words simultaneously rather than sequentially, it allowed for massive parallelization on GPUs, enabling trillion-parameter Large Language Models.
Who Invented It
Ashish Vaswani et al. (Google Brain & Research)
A team of eight researchers who transformed modern AI.
Applications
- Large Language Models (GPT)
- BERT
- AlphaFold
- Foundation Models
Key Papers
- Attention Is All You Need
Ashish Vaswani et al. · NeurIPS 2017
Videos
What are Transformers (Machine Learning Model)?
IBM Technology