Posters

Agents’ Last Exam is building the largest-scale, broadest-coverage agent evaluation benchmark to date, measuring performance on long-horizon, economically valuable tasks with verifiable outcomes. Led by Berkeley RDI and 300+ industry experts, it now spans all 55 targeted sub-industries covering most major fields of professional work performed on a computer, with 1,500+ tasks collected toward a 5,000-task target, keeping scores objective, comparable, and meaningful across domains.

AgentAbstain is a systematic research framework and benchmark that evaluates whether large language model (LLM) agents know when to stop and refuse unsafe, contradictory, or unresolvable tasks instead of blindly executing tool actions.

CollusionBench is an open-source framework for detecting agentic collusion by probing agents’ internal representations, not just their outputs, using a “collusion feature dictionary” that catalogs internal signatures of sub-behaviors like covert signaling or coordination planning. This lets it flag collusive reasoning even when an agent’s visible behavior looks benign.

Agent benchmarks silently degrade over time as tasks drift, verifiers overfit, and resource budgets shift, and the current fix relies on costly human re-annotation. Terminal Bench 2.1 (Continuous Benchmark Validation) offers a reproducible pipeline that combines multiple evidence sources (GitHub issues, fix-branch commits, writeups, trial traces) to automatically score task brokenness and diagnose failure causes, recovering 80% of human-identified broken tasks on a TB2.0 to TB2.1 replay and packaged as a reusable audit harness for any benchmark.

FutureSim is a live backtesting simulation that tests how well models adapt at test time to new information and forecast outcomes. Each run spans a 90-day simulated period, involving 5,000+ tool calls and over 12 hours of runtime per model.

HalluWorld is an extensible benchmark that defines hallucination against a fully specified reference world, allowing automatic labeling across synthetic environments like gridworlds, chess, and terminal tasks, rather than relying on human annotation or memorized references. Testing frontier models across these settings reveals that direct perceptual hallucination is largely solved, while multi-step state tracking, causal simulation, and knowing when to abstain remain persistent, distinct failure modes not fixed by extended thinking.

MBABench (FKA: BizBench) addresses the lack of a rigorous open benchmark for AI agents on ambiguous, end-to-end white-collar work. A prototype of 196 financial modeling tasks shows even top models score just 52.4/100, validating the approach, but scaling into an open, expert-verified benchmark across business domains requires domain experts to anonymize proprietary data and build professional-grade reference solutions.

MedPAIR is a benchmark testing whether LLMs identify clinically relevant information the same way physician trainees do, not just whether they get medical questions right. Using around 1,300 QA pairs annotated by physician trainees for sentence-level relevance, it compares physician and LLM relevance judgments and tests whether filtering out “irrelevant” content improves accuracy for either group, with all annotations and outputs released publicly

Paperena is a long-horizon environment for evaluating “AI scientists,” addressing the risk of AI-generated research slop by testing agents on realistic, extended research cycles (writing, reviewing, revising) rather than short, static tasks. It will measure factuality, reproducibility, novelty, and community behavior, producing open infrastructure, a public leaderboard, and released data to steer AI scientists toward valid and trustworthy research.

SciHarbor is a community benchmark platform that tests AI agents on real scientific workflows, using a unified Sci-MCP server to wrap domain simulators (OpenFOAM, GROMACS, AlphaFold, etc.) with tasks defined and scored by domain scientists, extending in a later phase to physical lab experiments at UMN CEMS. It aims to do for scientific discovery what SWE-bench and TerminalBench did for coding: build the evaluation infrastructure needed to measure and drive real progress.

SlopCodeBench evaluates coding agents the way real software actually gets built: through repeated requirement changes and extensions. Each problem is a sequence of checkpoints — the agent implements an initial version, then extends its own solution as new requirements arrive. Evaluation is black-box: only a CLI or API contract is given, with no prescribed architecture, function signatures, or module boundaries, so early design decisions compound across the run. Beyond correctness, we measure code erosion — verbosity, dead branches, and redundant structure — to surface the agents that stay clean under sustained change instead of patching their way into slop.

STELLA is a benchmark that tests how LLMs handle vulnerable patients (suicidal crisis, eating disorder minimization, domestic abuse) across long, multi-turn conversations, with a paper on MedRxiv and a live leaderboard scoring 25+ frontier models. It reveals that some models get less safe as a crisis conversation continues, and that the safest models often sacrifice real clinical helpfulness, a tradeoff no other benchmark captures.

Terminal-Bench Science is a benchmark for evaluating AI agents on real computational workflows from scientific research. It builds on Terminal-Bench, which has been adopted by frontier labs including Anthropic, OpenAI, and Google DeepMind and has helped drive progress in AI agents on software engineering tasks by defining what those labs measure and optimize for. Terminal-Bench Science brings the same approach to the natural sciences.

Scaling laws for pretraining tell you how to optimally allocate compute across your model and dataset size. With Train-to-Test (T²) scaling laws, we show that once you account for test-time compute, the optimal pretraining strategy shifts radically toward training smaller models for hundreds or thousands of times longer.