Sierra

Contribution to AI

What may outlast the company itself is a measuring stick. τ-bench, published in June 2024 with its code and data released openly, dropped a language model into a simulated customer-service conversation with a simulated user, gave it domain-specific API tools and a written policy document, and scored it by comparing the final state of the database against an annotated goal state — an evaluation of what an agent did rather than what it said. Its pass^k metric ran the same task repeatedly and counted only the trials that all succeeded, which exposed inconsistency as the central failure mode: state-of-the-art function-calling agents finished under half the tasks, and fewer than a quarter of the retail tasks survived eight consecutive runs. Frontier labs now quote τ-bench figures when they launch models , so a customer-service vendor's internal benchmark became one of the numbers by which general-purpose systems are judged; a 2025 successor extended it to dual-control settings where the user must also act . The commercial shift was to the unit of sale. Charging only when the agent achieves a defined result rather than per seat or per message tied revenue to completed work, and rivals restructured their pricing to match. Alongside it came a published account of building, testing and releasing agents as versioned software, drawn from deployments handling Sonos setup, SiriusXM subscriptions and ADT alarms at consumer volume.

Drafted with AI and edited by hand (claude-opus-5, reviewed 2026-08).