Posters

Agents’ Last Exam is building the largest-scale, broadest-coverage agent evaluation benchmark to date, measuring performance on long-horizon, economically valuable tasks with verifiable outcomes. Led by Berkeley RDI and 300+ industry experts, it now spans all 55 targeted sub-industries covering most major fields of professional work performed on a computer, with 1,500+ tasks collected toward a 5,000-task target, keeping scores objective, comparable, and meaningful across domains.

AgentAbstain is a systematic research framework and benchmark that evaluates whether large language model (LLM) agents know when to stop and refuse unsafe, contradictory, or unresolvable tasks instead of blindly executing tool actions.

CollusionBench is an open-source framework for detecting agentic collusion by probing agents’ internal representations, not just their outputs, using a “collusion feature dictionary” that catalogs internal signatures of sub-behaviors like covert signaling or coordination planning. This lets it flag collusive reasoning even when an agent’s visible behavior looks benign.

Agent benchmarks silently degrade over time as tasks drift, verifiers overfit, and resource budgets shift, and the current fix relies on costly human re-annotation. Terminal Bench 2.1 (Continuous Benchmark Validation) offers a reproducible pipeline that combines multiple evidence sources (GitHub issues, fix-branch commits, writeups, trial traces) to automatically score task brokenness and diagnose failure causes, recovering 80% of human-identified broken tasks on a TB2.0 to TB2.1 replay and packaged as a reusable audit harness for any benchmark.

FutureSim is a live backtesting simulation that tests how well models adapt at test time to new information and forecast outcomes. Each run spans a 90-day simulated period, involving 5,000+ tool calls and over 12 hours of runtime per model.

HalluWorld is an extensible benchmark that defines hallucination against a fully specified reference world, allowing automatic labeling across synthetic environments like gridworlds, chess, and terminal tasks, rather than relying on human annotation or memorized references. Testing frontier models across these settings reveals that direct perceptual hallucination is largely solved, while multi-step state tracking, causal simulation, and knowing when to abstain remain persistent, distinct failure modes not fixed by extended thinking.

MBABench (FKA: BizBench) addresses the lack of a rigorous open benchmark for AI agents on ambiguous, end-to-end white-collar work. A prototype of 196 financial modeling tasks shows even top models score just 52.4/100, validating the approach, but scaling into an open, expert-verified benchmark across business domains requires domain experts to anonymize proprietary data and build professional-grade reference solutions.

MedPAIR is a benchmark testing whether LLMs identify clinically relevant information the same way physician trainees do, not just whether they get medical questions right. Using around 1,300 QA pairs annotated by physician trainees for sentence-level relevance, it compares physician and LLM relevance judgments and tests whether filtering out “irrelevant” content improves accuracy for either group, with all annotations and outputs released publicly

Paperena is a long-horizon environment for evaluating “AI scientists,” addressing the risk of AI-generated research slop by testing agents on realistic, extended research cycles (writing, reviewing, revising) rather than short, static tasks. It will measure factuality, reproducibility, novelty, and community behavior, producing open infrastructure, a public leaderboard, and released data to steer AI scientists toward valid and trustworthy research.

PhantomEnvironments are synthetic RL training environments of fictional Wikipedia: produced entirely by rules and templates, fully verifiable, with no human annotation and no frontier-model distillation. They contain multi-hop questions over fictional social graphs, like “Who is the father of the sister of Alice?” RL fine-tuning in these strikingly simple environments trains LLM search agents that transfer to real-world multi-hop search, competitive with real-world training environments. Fiction leaves nothing to memorize, so the agent has to learn to search—and that skill is what transfers. Agents also generalize to unseen fictional universes and learn to scale their search budget with question difficulty, suggesting agentic behavior emerges from environment interaction alone. Ablations show linear hop questions carry nearly all of the transfer, while constraint questions hurt by teaching agents to paste the whole question as a single query.

SciHarbor is a community benchmark platform that tests AI agents on real scientific workflows, using a unified Sci-MCP server to wrap domain simulators (OpenFOAM, GROMACS, AlphaFold, etc.) with tasks defined and scored by domain scientists, extending in a later phase to physical lab experiments at UMN CEMS. It aims to do for scientific discovery what SWE-bench and TerminalBench did for coding: build the evaluation infrastructure needed to measure and drive real progress.

SlopCodeBench evaluates coding agents the way real software actually gets built: through repeated requirement changes and extensions. Each problem is a sequence of checkpoints — the agent implements an initial version, then extends its own solution as new requirements arrive. Evaluation is black-box: only a CLI or API contract is given, with no prescribed architecture, function signatures, or module boundaries, so early design decisions compound across the run. Beyond correctness, we measure code erosion — verbosity, dead branches, and redundant structure — to surface the agents that stay clean under sustained change instead of patching their way into slop.

SREGym is a benchmark that throws AI agents into live, realistic production failures and measures whether they can actually diagnose and fix them.

STELLA is a benchmark that tests how LLMs handle vulnerable patients (suicidal crisis, eating disorder minimization, domestic abuse) across long, multi-turn conversations, with a paper on MedRxiv and a live leaderboard scoring 25+ frontier models. It reveals that some models get less safe as a crisis conversation continues, and that the safest models often sacrifice real clinical helpfulness, a tradeoff no other benchmark captures.

Terminal-Bench Science is a benchmark for evaluating AI agents on real computational workflows from scientific research. It builds on Terminal-Bench, which has been adopted by frontier labs including Anthropic, OpenAI, and Google DeepMind and has helped drive progress in AI agents on software engineering tasks by defining what those labs measure and optimize for. Terminal-Bench Science brings the same approach to the natural sciences.

Scaling laws for pretraining tell you how to optimally allocate compute across your model and dataset size. With Train-to-Test (T²) scaling laws, we show that once you account for test-time compute, the optimal pretraining strategy shifts radically toward training smaller models for hundreds or thousands of times longer.