EleutherAI

Contribution to AI

When GPT-3 appeared in 2020 it was described in a paper and reachable through an interface, and that combination made a particular kind of research impossible: you cannot take apart what you cannot hold. The collective that formed in response was a Discord server, and it set out to rebuild something comparable in the open. The Pile came first and was the more consequential piece. Assembling over 800 gigabytes of text from twenty-two named sources and documenting what went into each made training data an object of study rather than an unexamined input — arguments about memorisation, contamination and licensing all need a corpus somebody is willing to describe. The GPT-Neo and GPT-J models that followed were not the largest available, but they were the largest anyone could download and dissect. Pythia was the sharper idea: a suite of models trained on identical data in identical order, with checkpoints saved throughout training and released alongside the finished weights. That turned questions about when a capability emerges during training from speculation into experiments. The broader demonstration was organisational — that volunteers with donated compute could work at a scale everyone had assumed belonged to corporate laboratories.