Agenda
October 8, 2026
San Francisco
The fastest way to improve AI itself may be to first improve how we measure it. This session examines the current state of evaluation and what a science of AI measurement could look like – from evaluating the evaluators and how we verify that a benchmark measures what it claims to, to the role of community, open development, and better tooling in making measurement itself more rigorous and reproducible.
What does rigorous, trustworthy evaluation have to become once models are savvy enough to recognize the test itself? This fireside chat explores the gap between what a benchmark says and what’s actually true, why so many of the field’s most trusted AI evaluations turn out to be easier to game than anyone expected, and what it takes to build ones that hold up under real scrutiny. When closing technical loopholes is only half the job, domain-specific human expertise becomes irreplaceable, raising broader questions about where the future of AI and human work actually headed and what gets lost when the conversation stays narrowly focused on scores.
As agents take on work that unfolds over hours or days rather than single turns, the old scorecard of simply getting the right answer stops being enough. This panel digs into the mechanics of building evaluation for exactly that regime: locating where the real bottleneck sits; grading outputs too complex for a straightforward pass/fail test; creating environments that don’t hand an agent more surface area to game the longer a task runs; and handling a system that keeps adapting after you’ve already scored it.
Designing reliable benchmarks is a fundamental research challenge. From evaluating research workflows across scientific domains, to studying different feedback protocols in open-ended legal work, to understanding how coding agents degrade over long-horizon iterative tasks, hear from the researchers building the benchmarks to close the evaluation gap in agentic AI.
