Yahoo!

Contribution to AI

Two pieces of infrastructure built for internal use ended up mattering more outside the company than in. Doug Cutting was hired in January 2006 to work on Hadoop, the first research cluster ran that March, and in May 2008 a 910-node Yahoo cluster sorted a terabyte in 209 seconds — the first time an open-source, Java-based program took that benchmark. Distributed storage and batch computation over commodity hardware stopped being something only Google could do, and the data pipelines that later fed machine learning at scale were assembled on it. The other contribution was data itself. Through the Webscope programme, Yahoo Research published corpora its products happened to generate: 4.4 million Yahoo Answers question-and-answer pairs, which became a routine text-classification benchmark; the graded relevance judgements behind the 2010 Learning to Rank Challenge, the first realistic web-ranking data anyone outside a search company could train on; and in 2014 the hundred million Creative Commons Flickr photos and videos of YFCC100M, still among the largest openly licensed multimedia collections and a source for later image-text training. Andrei Broder's group named computational advertising and framed ad placement as a matching problem to be learned rather than sold by hand. And the CAPTCHA exists because Yahoo needed to stop automated account signups in 2000 — an entire subfield defined by deliberately choosing tasks machine vision could not yet pass.

Drafted with AI and edited by hand (claude-opus-5, reviewed 2026-08).