Loading timeline…
20182010s
GPT Series (GPT-1, GPT-2)
The first generation of the Generative Pre-trained Transformer family.
Why It Was Important
OpenAI's approach utilized the Transformer 'decoder' to autoregressively predict the next word in massive text datasets. GPT-2 (1.5B parameters) famously generated coherent synthetic essays so well that OpenAI briefly refused to release the full model out of fear of mass disinformation styling.
Who Invented It
Alec Radford et al. (OpenAI)
The OpenAI team scaling unsupervised language modeling.
Applications
- Text Generation
- Zero-shot Learning
- Synthetic Media
Key Papers
- Improving Language Understanding by Generative Pre-Training
Alec Radford et al. · OpenAI · 2018
- Language Models are Unsupervised Multitask Learners
Alec Radford et al. · OpenAI · 2019