AI agent evals: frameworks and benchmarks

Princeton University (SWE-bench team)

Given a real Python repository and a GitHub issue, the agent must edit the code so the issue is fixed and the hidden tests pass.

OpenAI with the SWE-bench authors (Princeton)

The agent must resolve real GitHub issues in Python repositories, using a subset of SWE-bench that human engineers screened for fair tests and clear issue text.

SWE-bench team (Princeton University and collaborators)

The agent must fix bugs in user facing JavaScript libraries where the issue includes visual elements such as screenshots.

Scale AI

The agent must complete long horizon bug fixes and feature work in real business, B2B and developer tool codebases.

OpenAI

The agent must complete real paid Upwork software tasks, either implementing fixes and features or choosing the best proposal as a manager.

Stanford University and Laude Institute (Harbor team)

The agent must complete hard, realistic tasks inside a containerized terminal, from coding and system administration to science and ML work.

XLANG Lab, University of Hong Kong (with Salesforce Research, CMU, University of Waterloo)

The agent must complete open ended tasks on a real desktop computer across web and desktop apps, operating the screen with mouse and keyboard.

XLANG Lab, University of Hong Kong and collaborators

The agent must carry out long, realistic computer workflows across apps and self hosted websites, often taking a human more than an hour.

Carnegie Mellon University

The agent must complete long, realistic tasks on self hosted websites such as a shop, a forum, GitLab and a content manager.

OpenAI

The agent must persistently browse the internet to find hard to locate, entangled facts with short, checkable answers.

HKUST NLP

The agent must finish multi step workflows across many real apps, such as email, calendars, Kubernetes and BigQuery, mostly through MCP servers.

Salesforce AI Research

The agent must solve hard tasks by calling real world MCP servers in areas like maps, GitHub, finance, 3D design, browser automation and web search.

Accenture

The agent must plan and run multi step, cross tool workflows on live MCP servers from fuzzy instructions that do not name the tools.

Stony Brook University NLP

The agent must write interactive code against 457 APIs across 9 simulated everyday apps to complete tasks for simulated users.

UC Berkeley (Gorilla LLM team)

The model must call functions and tools correctly, including multi turn use, memory, web search and avoiding hallucinated calls.

Sierra

The agent must serve a simulated customer over a conversation, using domain tools and following a written company policy.

OpenAI

The agent must work through Kaggle machine learning competitions end to end, preparing data, training models and submitting predictions.

Center for AI Safety and Scale AI

The model or agent must answer expert level, closed ended academic questions across dozens of subjects, some with images.

Harbor and Terminal-Bench team (Stanford University, Laude Institute)

The agent must solve a compact, hard mix of tasks drawn from many existing agent benchmarks, run through one standard harness.

Meta FAIR, Hugging Face and AutoGPT

The assistant must answer real world questions that need reasoning, web browsing, file handling and tool use, each with one short correct answer.

METR

Agents attempt software, ML and cybersecurity tasks of known human duration, and the result is the task length at which they succeed half the time.

OpenAI

The model must produce real work deliverables, such as a legal brief or care plan, for 44 occupations across the top 9 US GDP sectors.

UC Berkeley RDI and RDI Foundation

The agent must complete long, economically valuable professional workflows on a computer, often in real industry software, with verifiable outcomes.

Mercor

The agent must complete long, cross application tasks written by investment banking analysts, management consultants and corporate lawyers inside realistic file and tool environments.