AI21 Labs

Contribution to AI

A second source for models at GPT-3's scale appeared within a year of GPT-3 itself, and that mattered more than any benchmark margin. Jurassic-1 Jumbo, released in August 2021 at 178 billion parameters, came out level with GPT-3 averaged across zero-shot tasks and was reachable through a studio where developers could train custom versions, at a moment when a model of that size otherwise meant a waiting list at one company. Its tokenizer was the sharper departure: a vocabulary of roughly 250,000 items that included multi-word expressions and named entities rather than words and word-pieces alone, so the same passage cost fewer tokens to represent and fewer steps to generate. The lab's papers then argued against the monolithic model. The 2022 MRKL proposal described a language model routing queries out to calculators, databases and other discrete modules, the arrangement that tool-using agents later standardised, and the in-context retrieval work showed that simply prepending retrieved passages to the prompt improved language modelling with off-the-shelf retrievers and no retraining. Jamba, in March 2024, was the first Mamba-style state-space model built at production scale, interleaving state-space layers with attention and mixture-of-experts routing, carrying a 256,000-token context and released under an Apache licence — evidence that the memory cost of long inputs was an architectural decision rather than a fixed property of language models.

Drafted with AI and edited by hand (claude-opus-5, reviewed 2026-08).