Posters

AgentAbstain is a systematic evaluation framework for agentic abstention: whether tool-using agents know when not to act, before or during execution. Its benchmark comprises 263 task pairs, each a should-act task and a perturbed should-abstain variant, across 42 MCP sandboxes; its AbstainGen pipeline generates environments and paired tasks on demand to avoid data contamination. Across 17 frontier models, none exceeds 60% paired accuracy, and abstention capability is largely independent of task-solving capability. We identify agent-specific failure patterns such as post-hoc abstention, where agents commit irreversible actions and only then refuse.

Agents’ Last Exam is building the largest-scale, broadest-coverage agent evaluation benchmark to date, measuring performance on long-horizon, economically valuable tasks with verifiable outcomes. Led by Berkeley RDI and 300+ industry experts, it now spans all 55 targeted sub-industries covering most major fields of professional work performed on a computer, with 1,500+ tasks collected toward a 5,000-task target, keeping scores objective, comparable, and meaningful across domains.

ApprenticeBench is the first end-to-end benchmark of how well an agent learns a real job, spanning computer use, continual learning, and long-horizon agentic capabilities. It mirrors how human employees gain expertise: through onboarding, observing the work, practicing under review, and eventually working independently with occasional feedback. Our results show progress across model generations in using software and learning company rules, but even the strongest agents still face challenges in reliability, efficiency, and communication.

AutoResearchExam is a benchmark of open-ended ML research and engineering tasks, built to drive progress toward recursive self-improvement (RSI). Agents iteratively experiment with validation feedback over 24-hour research runs and are scored on fast, sustained progress that generalizes to hidden tests. Tracking that progress reveals how model rankings shift as research unfolds, making the case for evaluating research capability across a continuous time budget.

CollusionBench is an open-source framework for detecting agentic collusion by probing agents’ internal representations, not just their outputs, using a “collusion feature dictionary” that catalogs internal signatures of sub-behaviors like covert signaling or coordination planning. This lets it flag collusive reasoning even when an agent’s visible behavior looks benign.

FutureSim is a live backtesting simulation that tests how well models adapt at test time to new information and forecast outcomes. Each run spans a 90-day simulated period, involving 5,000+ tool calls and over 12 hours of runtime per model.

HalluWorld is an extensible benchmark that defines hallucination against a fully specified reference world, allowing automatic labeling across synthetic environments like gridworlds, chess, and terminal tasks, rather than relying on human annotation or memorized references. Testing frontier models across these settings reveals that direct perceptual hallucination is largely solved, while multi-step state tracking, causal simulation, and knowing when to abstain remain persistent, distinct failure modes not fixed by extended thinking.

As AI agents take on more complex tasks, we need to know whether human oversight can keep pace. HumanOversight Bench uses scenarios grounded in real-world failures to measure how well people catch AI coding agents’ mistakes and how repeated approval requests affect their attention and judgment.

MBABench (FKA: BizBench) addresses the lack of a rigorous open benchmark for AI agents on ambiguous, end-to-end white-collar work. A prototype of 196 financial modeling tasks shows even top models score just 52.4/100, validating the approach, but scaling into an open, expert-verified benchmark across business domains requires domain experts to anonymize proprietary data and build professional-grade reference solutions.

MedPAIR is a benchmark testing whether LLMs identify clinically relevant information the same way physician trainees do, not just whether they get medical questions right. Using around 1,300 QA pairs annotated by physician trainees for sentence-level relevance, it compares physician and LLM relevance judgments and tests whether filtering out “irrelevant” content improves accuracy for either group, with all annotations and outputs released publicly

ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. We construct a live OpenTelemetry-instrumented microservice system, exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access. We then curate 1,079 realistic root cause analysis tasks by systematically varying report specificity, time-to-detection, and co-occurring fault scenarios.

Paperena is a long-horizon environment for evaluating “AI scientists,” addressing the risk of AI-generated research slop by testing agents on realistic, extended research cycles (writing, reviewing, revising) rather than short, static tasks. It will measure factuality, reproducibility, novelty, and community behavior, producing open infrastructure, a public leaderboard, and released data to steer AI scientists toward valid and trustworthy research.

PhantomEnvironments are synthetic RL training environments of fictional Wikipedia: produced entirely by rules and templates, fully verifiable, with no human annotation and no frontier-model distillation. They contain multi-hop questions over fictional social graphs, like “Who is the father of the sister of Alice?” RL fine-tuning in these strikingly simple environments trains LLM search agents that transfer to real-world multi-hop search, competitive with real-world training environments. Fiction leaves nothing to memorize, so the agent has to learn to search—and that skill is what transfers. Agents also generalize to unseen fictional universes and learn to scale their search budget with question difficulty, suggesting agentic behavior emerges from environment interaction alone. Ablations show linear hop questions carry nearly all of the transfer, while constraint questions hurt by teaching agents to paste the whole question as a single query.

Benchmarking Recursive Self-Improvement through Executable Research
It evaluates whether an AI agent can improve itself over a long horizon and generalize to unseen data. Hours of autonomous experimentation on the method that solves the task or the harness that drives a frozen model, then one final run on a hidden test set.

SciVeri-Bench is a deploy-first, scientist-driven benchmark that evaluates AI agents’ ability to perform scientific verification as human domain scientists do on open-ended, novel scientific tasks. Our benchmark covers three verification tasks: scientist critique generation, scientist rubric generation, and scientist taste evaluation. To construct the benchmark, we develop an automated platform designed for domain scientists rather than AI researchers—a natural starting point for scientists who want to try AI agents.

As new AI models are released at an increasingly rapid pace, serving systems need to support them correctly and efficiently from day one. But bringing up a new model can require substantial engineering across model architectures, serving runtimes, hardware-specific kernels, and performance optimization. ServingBench studies whether coding agents can automate this process end-to-end, using 148 real-world model bring-up tasks across major serving engines and diverse accelerators (e.g., Nvidia / AMD GPUs and TPUs), and reveals a significant gap between generating code that works and delivering production-quality serving performance.

SlopCodeBench evaluates coding agents the way real software actually gets built: through repeated requirement changes and extensions. Each problem is a sequence of checkpoints — the agent implements an initial version, then extends its own solution as new requirements arrive. Evaluation is black-box: only a CLI or API contract is given, with no prescribed architecture, function signatures, or module boundaries, so early design decisions compound across the run. Beyond correctness, we measure code erosion — verbosity, dead branches, and redundant structure — to surface the agents that stay clean under sustained change instead of patching their way into slop.

SREGym is a benchmark that throws AI agents into live, realistic production failures and measures whether they can actually diagnose and fix them.

STELLA is a benchmark that tests how LLMs handle vulnerable patients (suicidal crisis, eating disorder minimization, domestic abuse) across long, multi-turn conversations, with a paper on MedRxiv and a live leaderboard scoring 25+ frontier models. It reveals that some models get less safe as a crisis conversation continues, and that the safest models often sacrifice real clinical helpfulness, a tradeoff no other benchmark captures.

Agent benchmarks silently degrade over time as tasks drift, verifiers overfit, and resource budgets shift, and the current fix relies on costly human re-annotation. Terminal Bench 2.1 (Continuous Benchmark Validation) offers a reproducible pipeline that combines multiple evidence sources (GitHub issues, fix-branch commits, writeups, trial traces) to automatically score task brokenness and diagnose failure causes, recovering 80% of human-identified broken tasks on a TB2.0 to TB2.1 replay and packaged as a reusable audit harness for any benchmark.

Terminal-Bench Science is a benchmark for evaluating AI agents on real computational workflows from scientific research. It builds on Terminal-Bench, which has been adopted by frontier labs including Anthropic, OpenAI, and Google DeepMind and has helped drive progress in AI agents on software engineering tasks by defining what those labs measure and optimize for. Terminal-Bench Science brings the same approach to the natural sciences.

TLAPS-Bench offers a service for formally proving the correctness of complex protocols and systems using TLA+ Proof System (TLAPS). It has been used to formally verify protocols/systems that were never proved before, and can be used (together with Specula) to formally verify production system code. It provides a rich, extensible, and scalable benchmark suite that poses significant challenges for frontier AI.

Scaling laws for pretraining tell you how to optimally allocate compute across your model and dataset size. With Train-to-Test (T²) scaling laws, we show that once you account for test-time compute, the optimal pretraining strategy shifts radically toward training smaller models for hundreds or thousands of times longer.