This background briefing guide was distributed to participants ahead of the Forum for Cooperation on AI (FCAI) dialogue on Sept. 15, 2026. FCAI dialogues follow Chatham House rule, so remarks are anonymized.
Artificial intelligence (AI) evaluations aim to measure AI models or systems under specified conditions for a particular capability or risk, or for systemic capabilities, risks, performance, or societal impact. Directed evaluations typically select a target and measure an observable proxy for it, aiming to minimize the construct validity gap, or the distance between the target and the proxy. Exploratory evaluations can also be used to anticipate future AI system properties, impacts, risks, use cases, or areas of uncertainty. Both evaluation types can serve as measurement tools for policymakers and regulators.
Governments and organizations of various kinds are building evaluation capacity simultaneously, and recent events gave that work new urgency. In June, Anthropic released Claude Mythos 5 and Claude Fable 5, models reportedly capable of discovering previously unknown software vulnerabilities and generating working exploits. Anthropic’s prerelease of these models through Project Glasswing, a collaborative security review process, prompted President Donald Trump to issue an executive order calling for a voluntary prerelease review framework. Weeks later, the administration moved to halt foreign nationals’ access to the model through an export-control directive, which was lifted by the end of June.
The following month, OpenAI models broke out of their test environment and hacked the Hugging Face platform to game model tests. In September, Google disclosed that its AI model, Gemini, breached three other companies during a cybersecurity model evaluation conducted by third-party firm Irregular in May.
General-purpose AI (GPAI) models have no defined task, envelope, or intended use to verify against; their utility is extraordinarily broad. The models are complex, opaque, and unpredictable, and evaluation practices are still forming. Certain capabilities and risks—cybersecurity offense, chemical, biological, radiological, and nuclear (CBRN) uplift, and misinformation—are widely recognized and subject to testing, but new capabilities are emerging faster than methods for evaluating even the consensus risks.
Evaluations can target any part of the model supply chain, at different development and deployment stages, or an entire AI system in its deployment context. Model safety evaluations focus on model outputs and tend to evaluate capabilities. By contrast, contextual safety evaluations assess a model as deployed: embedded in a particular product, wrapped in a particular set of safeguards, and used by a particular population. Beyond measuring what a model is capable of in isolation, a sociotechnical approach asks what happens when people use it: whether they can complete their intended tasks, whether outputs fit the context of use, and whether adversarial users can bypass safety restrictions. This framing measures both whether a behavior can be elicited and how often it arises under realistic conditions of use.
Definitions vary across developers, regulators, and adopters. Developers and regulators often use “capability evaluation” to determine whether an AI system is safe to deploy, whereas adopters may use the same term for the targeted measurement of positive capabilities. The EU AI Act references evaluations in various sections, some oriented to capabilities or performance and others to risks (see Appendix A).
Benchmark evaluations now report that GPAI systems perform at or above human-expert baselines on several academic question sets, including scores over 90% on undergraduate-level exams in subjects from chemistry to law (MMLU) and over 80% on graduate-level science questions (GPQA). But exceeding a human baseline on a fixed set of questions does not indicate expert-level competence, and performance degrades quickly as tasks lengthen. Model Evaluation and Threat Research (METR) measures a model’s “time horizon,” the length of task, measured by how long human experts take to complete it, at which a model succeeds at a given rate of reliability. Leading models reach 50% reliability on software tasks that take human experts roughly an hour and fall below 10% on tasks that take humans more than about four hours.
No evaluation can give the full picture. What an evaluation leaves out is itself a choice, as are its goal, scope, and interpretation of results. These choices reflect the perspective and motivation of its designers. Evaluations are therefore not neutral instruments: Deciding what to measure, how to measure it, and how to interpret results requires value judgment from evaluators.
How do AI evaluations work?
Evaluation methods for AI did not begin with current frontier models. They build upon software testing methods but require larger scale and scope. Predictive machine learning systems have long been evaluated against a known task distribution. For example, a model trained to classify loan applications or flag tumors has a defined input, a validation set drawn from the same distribution as its training data, and ground-truth labels. Even under those conditions, held-out performance proved an unreliable guide to deployment behavior.
Foundation models break the assumption of a defined task distribution. Since a GPAI model is trained on broad data and adapted to a wide range of downstream tasks, no held-out set represents deployment. Benchmarks therefore serve as proxies for capabilities that cannot be directly measured (the construct validity challenge). Measured capability also depends on how a model is prompted, what tools it is given, and how much compute it is allowed. An evaluation that does not attempt to elicit a model’s full capability shows a floor of what the model did under those particular conditions, not what it may be able to do. A later evaluator with better prompting may be able to find capabilities in the same model that the first one missed.
Agentic systems shift the unit of evaluation again: An agent completes multistep trajectories, has tool-calling capabilities, and acts on an environment.
Evaluations must assess the deployed AI system: the model and its tools, permissions, memory, and operating environment. The National Institute of Standards and Technology’s (NIST) TEVV-Athlon draft extends to agentic systems on this basis, and Singapore’s AI Safety Institute (AISI) frames the shift as testing systems rather than models.
Capability evaluations conducted in a testing environment can miss contextual factors of human-AI interaction in deployment settings. An evaluation may measure the likelihood of a system producing factually incorrect output, but the actual risk of people being deceived depends on the context of use, the users, and features of the application, such as whether AI-generated content is watermarked.
Methods for measuring risk in context exist. At the human interaction level, risk is typically measured through user studies and uplift studies comparing task outcomes with and without model access under test and control conditions. NIST’s Assessing Risks and Impacts of AI (ARIA) program uses three tiers: (1) model testing to assess whether a system performs as advertised, (2) red teaming to probe adversarial use, and (3) field testing to evaluate behavior in realistic scenarios with human users. ARIA field testing may involve several thousand participants interacting with AI applications across multiple sessions under test or control conditions, scored against a Contextual Robustness Index (CoRIx), which gives a suite of metrics to measure AI systems’ functionality across different deployment conditions. Singapore’s Global AI Assurance Pilot applied similar logic at the application layer by pairing assurance vendors with firms deploying live generative AI products.
At the systemic layer, methods are thinner, relying on post-market monitoring, incident reporting, and observational study rather than controlled testing—one reason the EU Code of Practice ties escalation of assessment depth to post-market and incident data rather than to prerelease testing alone. Capabilities may contribute to large-scale effects, but a capability evaluation cannot determine outcomes without examining systemic factors. AI-generated misinformation likely affects public trust in shared media, for instance, but capability metrics alone will not illustrate a trend without analysis of norms in information sharing or the existence of fact-checking institutions. Capability evaluation gives an early indication of potential downstream harms but requires context to determine risks of harm.
Approaches to model evaluation
Evaluators across both public and private sectors have differing relationships with developers and deployers, which affects their access to information and their objectivity and independence. There have been proposed scales for types of evaluations by level of independence (Figure A).
First-party evaluations are run in-house on an organization’s own models, data, or applications. The authors distinguish first-party limited, carried out by the team responsible for building and validating the system’s performance, from first-party expansive, carried out by a separate team dedicated to flaw detection but not responsible for building the system, such as a red team or internal compliance team.
Second-party evaluators have a contractual or obligatory relationship with the provider or purchaser. This could include a developer providing a customer with model access or a contracted organization evaluating on a developer’s behalf. Their work is usually more formal because the contractual terms lay out the evaluation. Second-party limited evaluations are carried out by a specific contracted party and are limited in time and scope; second-party expansive evaluations are carried out by various contracted parties. Paid evaluation consultancies and potential purchasers of an AI system are other examples.
Third-party evaluators have no existing contractual relationship with the developers or deployers. The paper distinguishes between preapproved third-party evaluations, where the provider vets evaluators and provides access beforehand; third-party limited evaluations, which are carried out without safe harbor; and third-party expansive evaluations, where third parties have safe harbor and coordinated flaw-reporting infrastructure.
Third-party evaluations can also be categorized by who decides to conduct them.
They can be evaluatee-driven, where the company being evaluated hires the evaluator and may set a specific objective, ask for evaluations certifying compliance with an independent standard, or grant a wide remit to conduct exploratory evaluations; government-driven, where a government or independent regulator appoints an evaluator or evaluates directly; or independent, where academics, journalists, or civil society organizations evaluate based on their own priorities, with or without the developer’s oversight.
Some jurisdictions require risk-assessment evaluations for AI systems in certain contexts. For example, New York City’s Local Law 144 requires annual bias audits by an independent auditor for automated employment-decision tools.
Different evaluations require varying levels of system access, from public queries, probabilities, gradients, weights and activations, and fine-tuning access to outside-the-box and unrestricted access.
Designing an AI evaluation
AI evaluation designers make decisions about what to measure, how to measure, and how to interpret their results. NIST’s Center for AI Standards and Innovation’s (CAISI) initial public draft of NIST AI 800-2, “Practices for Automated Benchmark Evaluations of Language Models” recommends that designers ask two questions: How will the measurements be used, and what should be measured? The answer to the first question informs the second, and designers then determine and record the measurement construct for selected use cases.
Potential uses of evaluation results include informing or assessing development decisions; assessing whether a system is fit to use for a specific scenario; comparing systems for deployment readiness; assessing deployment configuration efficacy; validating that a system has been deployed and configured correctly; and informing predictions of real-world impacts. Depending on the objective, an evaluation may measure a model, a system, or an individual component or algorithm. Table 1 presents examples of goals and associated measurement constructs.
NIST released the initial public draft of AI 200-2, the TEVV-Athlon Framework for Evaluating AI Systems, on Aug. 7, with comments due Oct. 6. Where AI 800-2 addresses automated benchmark evaluations end to end, from defining objectives through analyzing and reporting results, AI 200-2 is a four-stage method for building customized assessments from an organization’s TEVV objectives rather than a standard list of system tests. NIST describes it as extensible across statistical machine learning models, LLMs, multimodal models, and agentic systems.
Evaluation methods
Model safety evaluations, which tend to measure capabilities, are often conducted through capability testing or benchmarking (shared tests used to compare how well AI systems perform specific tasks). Contextual safety evaluations, which assess a model as deployed, with its safeguards and users, may include red-teaming (a structured testing effort, often adopting adversarial methods, to find flaws and vulnerabilities in an AI system) or uplift studies (comparison of how testers complete a task with and without model access to evaluate whether the model provided meaningful assistance).
Red-teaming often relies on a manual human-led process, which takes longer but allows teams to adapt evaluations to the specific models and systems at hand.
Some benchmark evaluations run automated, without a human in the loop. Human oversight generally increases accuracy, but automated benchmarks may be less costly, quicker, and easier to scale. Since benchmarks are built by many parties under no common methodological standard, they vary significantly in quality, scope, validity, and robustness. Some researchers have argued for better deprecation practices for benchmarks, given the static nature of the evaluations. Some frontier model developers have also used automated red-teaming. Automated evaluations have been criticized as “superficial” and “unreliable,” and for overestimating model utility.
Evaluations of harmful outcomes test different outcomes than capability tests. Typically, evaluators will run prompts from a harm taxonomy against the model and score whether the model complied, and if the result was actionable and could result in a harmful outcome. Scores do not always reflect whether it was a single-turn prompt or a larger number of attempts that resulted in actionable outputs, or if prompting changed over time. When CAISI red-teamed an agent in collaboration with the U.K. AI Security Institute using attacks developed specifically for that model rather than known baselines, the measured attack success rate rose from 11% to 81%.
Evaluation may also include model elicitation, or “technical work to systematically enhance a model’s capabilities, propensities, affordances, and/or effects, thereby facilitating an accurate measurement of the full range of its capabilities, propensities, affordances, and/or effects that can likely be attained” to test a fuller extent of model capabilities.
New York City’s Local Law 144 and the EU AI Act both use disaggregated evaluation methods to measure bias and disparate performance. Known limits to this evaluation approach include that subgroups must be specified in advance and that demographic information needed for disaggregation can be unavailable or legally restricted.
United States
NIST’s AI Risk Management Framework (RMF) remains the reference vocabulary for many evaluations structuring organizational processes into four functions: govern, map, measure, and manage. Its “measure” function establishes test, evaluation, verification, and validation (TEVV) practices: Metrics should be documented, evaluation approaches themselves assessed for validity and reliability, and independent parties involved where appropriate. NIST also published a Generative AI Profile expanding the RMF to allow NIST to run open evaluation challenges, including the 2026 GenAI Text Challenge. A March memorandum of understanding between the General Services Administration and CAISI applies NIST’s evaluation science research to federal procurement decisions.
CAISI runs cyber and national security testing of frontier models, and NIST describes the center as the “industry’s primary point of contact within the U.S. government to facilitate testing.” In May, CAISI signed predeployment evaluation agreements with Google DeepMind, Microsoft, and xAI. Microsoft signed a parallel agreement with the U.K. AI Security Institute the same day covering predeployment evaluation, joint methodology, and information sharing on national security-relevant capabilities. CAISI aims to lead evaluations of U.S. and adversary AI systems, assess security vulnerabilities and malign foreign influence arising from the use of adversaries’ AI systems, and coordinate evaluation methods across the U.S. government.
Executive Order 14409, “Promoting Advanced Artificial Intelligence Innovation and Security,” signed June 2, creates a voluntary prerelease federal access window for “covered frontier models” of up to 30 days before developers release them to other trusted partners. It also directs the Treasury Department, with the national cyber director, National Security Agency (NSA), and Cybersecurity and Infrastructure Security Agency (CISA), to establish an AI cybersecurity clearinghouse coordinating the discovery, validation, and remediation of software vulnerabilities, and it explicitly declines to authorize mandatory licensing, preclearance, or permitting.
Model designations are delegated to the NSA director, in consultation with the national cyber director, the assistant to the president for science and technology, and the director of CISA. Since the benchmarking and thresholds behind those designations are classified, the public will not necessarily know which models are in scope or why. The vetting criteria for “trusted partners” eligible for early access are undefined. According to reporting, the White House convened AI companies in early August to review the completed framework, but the text has not been published, and the order’s 60-day deadline passed without a public deliverable.
The Mythos and Fable episode shows how evaluation results are informing U.S. policy on an ad hoc basis and how little of the underlying evidence reaches the public.
Anthropic’s Project Glasswing was a private analogue to the order’s trusted-partner structure, gating the unreleased models to vetted critical infrastructure operators before public release. After the public release of Fable 5 and Mythos 5 on June 9, the Commerce Department issued an export-control directive to Anthropic, requiring suspension of access for all foreign nationals. Anthropic said at the time that it had not received specific details explaining the national security concern. Commerce later cleared Mythos 5 for a limited group of vetted U.S. institutions and agencies, then terminated the directive.
At several points—the Project Glasswing admission criteria, the initial directive, and the subsequent clearance—decisions about model access were made without public notice or explanation. Anthropic disputed the severity finding, arguing the capability was comparable to what other publicly available models could already do, but the disagreement could not be adjudicated publicly because no test or threshold was published. Evaluation results increasingly drive consequential government action even as the evaluations and thresholds behind those judgments become more sensitive, and so less available to those affected by them.
States are also legislating on evaluations. California Senate Bill 53, signed in September 2025, requires large frontier developers to report, before deploying a new or substantially modified frontier model, summaries of catastrophic risk assessments, their results, and the extent of evaluator involvement, with reporting starting in January 2027. The law allows but does not require a third-party evaluator, though it mandates disclosure of whatever involvement occurred. This may encourage growth of a marketplace and standards for evaluation.
European Union
The AI Act’s evaluation obligations sit mainly in Article 55, which applies to providers of GPAI models with systemic risk. Three of its four obligations involve assessment and measurement. Providers must perform model evaluation in accordance with standardized protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing to identify and mitigate systemic risk; assess and mitigate systemic risks at union level, including their sources across development, market placement, and use; and track, document, and report serious incidents and corrective measures to the AI Office and national competent authorities. Until a harmonized standard is published, providers may rely on codes of practice under Article 56 to demonstrate compliance.
The General-Purpose AI Code of Practice adds detail. Measure 1.2 of its Safety and Security chapter distinguishes continuous lighter-touch assessment from systemic risk assessments. Signatories run lighter-touch evaluations, such as automated evaluations, at triggers defined by time, training or inference compute, development stage, user access, or model affordances; monitor for serious incidents post-market; and escalate the breadth or depth of assessment, based on those results. A full assessment is required before a model is placed on the market. The AI Office convened a July 2026 workshop on independence and qualification requirements for external evaluators under the code and maintains the Network of Evaluators.
The commission’s Action Plan on Cybersecurity and Artificial Intelligence, published July 7, 2026, sets nine key actions across 2026-27, including criteria for independent third-party evaluators by 2027 to support predeployment evaluation of frontier models, a dedicated European capacity to evaluate AI models with a cybersecurity focus by 2027, a European Blueprint with the European Union Agency for Cybersecurity (ENISA) for structured access to advanced models, and an ENISA/Joint Research Centre secure testing platform by the end of 2026. European trade press tied the initiative directly to Mythos’ vulnerability-detection capability and U.S. access restrictions on foreign nationals.
United Kingdom
The U.K. AI Security Institute developed Inspect, an open-source, MIT-licensed Python framework for reproducible model evaluations. Inspect is complemented by Inspect Evals, a separate repository of over 200 prebuilt evaluations. Both have been adopted by METR, Apollo Research, CAISI, other national AI safety institutes (AISIs), and major frontier labs.
The institute’s inaugural “Frontier AI Trends Report,” published in December 2025, aggregates two years of evaluations across more than 30 frontier models. In a July 2026 test of GPT-5.4, GPT-5.5, GPT-5.6 Sol, Claude Opus 4.7, and Claude Mythos Preview across 475 runs each in cybersecurity capability evaluations, all five models exhibited cheating behavior. The cheating rate did not track capability, and models described the behavior as wrong less than half the time, often omitting it from their reasoning. During one misconfigured evaluation with no possible solution, a model ran code on an external internet-hosted service in an apparent attempt to reach AISI’s own evaluation systems, triggering a security alert.
Japan
Japan’s AI Safety Institute published the Guide to Evaluation Perspectives on AI Safety (Version 1.20) in July 2026, updated and expanded specifically in light of the spread of AI agents, and a health care sector evaluation guide in April 2026. Its other evaluation resources include a red-teaming methodology guide (v1.10), an open-source AI safety evaluation tool, a fiscal year 2025 conformity assessment subworking group report, a data quality guidebook, and a July 2026 evaluation guide for AI in robotics.
Singapore
Singapore has contributed significantly to evaluating deployed applications as well as models. The Global AI Assurance Pilot, launched by the AI Verify Foundation and Infocomm Media Development Authority (IMDA) in February 2025, paired 17 organizations’ generative AI applications with 16 specialist testing firms; 33 organizations from 10 regions participated, and 16 of the 17 applications were already live in production. It became the ongoing Global AI Assurance Sandbox in July 2025, which has since supported testing of 30 applications across 14 sectors and 12 regions. The accompanying Starter Kit for Testing Large Language Model (LLM)-Based Applications covers five harms: hallucination, bias in decisionmaking, undesirable content, data leakage, and vulnerability to adversarial prompts. The starter kit pairs output testing of overall application behavior with component testing when outputs fall short. Both contribute to the AI Verify Foundation’s open-source toolkit, Project Moonshot.
Singapore AISI leads the testing and evaluation track under the AISIs network and has co-led three joint exercises. A multilingual exercise tested two open-weight models across 10 languages and the five harm categories above, using over 6,000 newly translated prompts assessed by both an LLM-as-a-judge approach and human annotation. The findings caution against treating a translated benchmark as a multilingual one.
A bilateral exercise with Korea’s AISI tested whether agents complete routine multistep workflows without leaking sensitive data. Across 12 tasks spanning enterprise productivity and customer service, agents completed tasks successfully while still exhibiting data-handling failures. A third, networkwide exercise extended this to fraud and cybersecurity, with Singapore leading the sensitive-information and fraud strand and the U.K. leading cybersecurity.
The Singapore Consensus on Global AI Safety Research Priorities, produced at the 2025 Singapore Conference on AI, organizes safety research on a defense-in-depth model into three categories: creating trustworthy systems (development), evaluating their risks (assessment), and monitoring and intervening after deployment (control). The 2026 edition adds a companion report on agentic risk management. IMDA and Singapore AISI also announced a partnership with Microsoft covering agentic AI safety, evaluation methods and benchmarks, and a forthcoming joint white paper on enabling governments and infrastructure operators to access frontier models responsibly.
Canada
The Canadian AI Safety Institute has published guidance on evaluation reporting and best practices. It is led by Innovation, Science and Economic Development Canada, draws on the National Research Council, and funds investigator-led work through the Canadian Institute for Advanced Research (CIFAR). It committed CA$1 million to the U.K.’s AI Security Institute’s Alignment Project in July 2025, and CIFAR announced the four resulting Canadian projects in February 2026.
Australia
Australia’s AI Safety Institute was formally established in early 2026 and, as of July, is actively testing unreleased frontier models, with the Australian Signals Directorate and the Commonwealth Scientific and Industrial Research Organisation. Like other AISIs, it is an advisory, not regulatory, body. Anthropic signed a memorandum of understanding with the Australian government on March 31, committing to share findings on emerging capabilities and participate in joint evaluations.
NAAIMES
The International Network of AI Safety Institutes was renamed the International Network for Advanced AI Measurement, Evaluation and Science (NAAIMES) in December 2025. It consists of AISIs from Australia, Canada, the European Union, France, Japan, Kenya, the Republic of Korea, Singapore, the United Kingdom, and the United States, with the U.K. as coordinator.
Recent outputs include:
- The first consensus list of key practices and open questions for automated AI evaluations (February 2026), reflecting CAISI’s draft NIST AI 800-2 work
- The first best-practice guidance on automated evaluations, published following the Seoul meeting on the margins of ICML 2026 and aimed at the growing ecosystem of third-party evaluators
- A joint agentic-safety testing exercise, split into sensitive-information leakage and fraud (led by Singapore AISI) and cybersecurity (led by U.K. AISI)
- A joint testing report on multilingual guardrails (Singapore and Japan)
Independent testing organizations
METR, a nonprofit that evaluates frontier AI systems, runs dangerous-capability and misalignment evaluations, including a pilot conducted between February and March and published in May that spanned Anthropic, Google, Meta, and OpenAI (see also this 2026 safety analysis). Apollo Research, a public benefit organization specializing in evaluations for deceptive alignment, offers Watcher, a product that provides organizationwide monitoring and real-time controls for coding agents. Several firms work directly with frontier labs to evaluate model alignment and safety controls.
Benchmark and assurance providers also contribute. MLCommons, an AI engineering consortium, runs the AILuminate benchmark, which evaluates generative AI models across 12 hazard categories and produces a risk rating per hazard, using input from AI companies, academia, and civil society. Its AILuminate Global Assurance Program, announced Feb. 19 and backed by KPMG, Google, Microsoft, and Qualcomm, aims to evaluate AI reliability; it includes the global framework for region-, language-, and industry-specific benchmarks and a risk label, which gives procurement officers and policymakers legible markers of benchmark results.
The commercial marketplace for evaluation
Every major AI lab publishes evaluation reports and makes safety claims based on them. Beginning in September 2023, several companies started voluntarily publishing corporate protocols for how they evaluate models for risks. In May 2024, 16 companies agreed to publish these protocols as part of the Frontier AI Safety Commitments at the AI Seoul Summit; four more have signed on, bringing the total to 20. Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and Google DeepMind’s Frontier Safety Framework each define capability thresholds for cybersecurity offense, biological weapons assistance, and automated research and development threat models. Some safety policies also include autonomous replication, advanced persuasion, and deceptive alignment.
Alongside these consortium efforts, a commercial assurance market has developed to evaluate models. In 2024, the U.K.’s Department for Science, Innovation and Technology identified 524 firms supplying AI assurance services, including 84 specialized AI assurance companies, 225 AI developers, 182 diversified firms, and 33 in-house adopters, up from 29 specialized firms in its 2023 AI Sector Study. Its September 2025 Trusted Third-Party AI Assurance Roadmap projects the assurance market growing to £18.8 billion by 2035 if barriers are addressed, while noting that no fully active AI assurance certification schemes are yet issued by U.K. accredited organizations.
Established audit and professional firms are extending existing assurance practices to AI, including KPMG’s participation in MLCommons’ Global Assurance Program. Specialized safety and security evaluation firms sell adversarial testing as a product; governance and compliance platforms sell documentation, policy-mapping, and monitoring; and cloud and security platform vendors such as CrowdStrike and Microsoft are commercializing red-team capabilities. Most of this work is second party on the axis in Figure A below, since firms contract with developers, or with potential purchasers, to run evaluations under nondisclosure agreements.
Industry bodies
Industry has also taken on evaluation standard-setting. The Frontier Model Forum—whose corporate members include Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI—publishes technical reports proposing what third-party assessments should look like, noting that few organizations can conduct rigorous and secure assessments. Individual developers are designing evaluation standards, too, such as OpenAI’s May 2026 playbook for third-party evaluations, which specifies what an external evaluation report should disclose: the claim being tested, elicitation methods, validity checks, and checks for evaluation awareness.
Current challenges in AI evaluations
Previous sections describe a rapidly expanding evaluation landscape and growing methodological toolkit. Neither yet amounts to a reliable capacity to assess frontier AI systems before or after deployment. Brookings researchers have identified several challenges in current AI evaluation practices. Below, we examine questions on what evaluations should measure, who should conduct them, and what should happen to evaluation results.
What are the current limitations of benchmarks?
Benchmarks do not reflect deployment conditions
The International AI Safety Report calls the divergence between predeployment test results and real-world capabilities the “evaluation gap.” GPAI model generality leads to unpredictable emergence of new capabilities in deployment, and evaluators often have poor visibility or understanding of model workings and downstream applications, making it difficult to anticipate the risks a GPAI model will present, and therefore which to test for. Even an identified failure point is likely a lower bound, as bad actors continue to jailbreak models and exploit known vulnerabilities. Given the pace of new releases, long evaluations may be out of date once completed.
There is also often a lack of common understanding of the unit of measurement. For AI agents, it is often unclear whether the object of evaluation is a model, a tool-use configuration, or an end-to-end workflow, and results may differ between levels. General-purpose benchmarks compound this by averaging across tasks in ways that may fail to reflect domain- and context-specific performance, so a strong aggregate score can coexist with poor performance in a specific setting. Accuracy-focused metrics can also obscure cost-performance trade-offs, omitting the compute, latency, or expense required to complete a task at a usable rate. Researchers have moved to improve methods and expand metrics tested, including on economically valuable and remote labor tasks.
Well-targeted benchmarks can produce unreliable measurements
Even a benchmark measuring the right construct may measure it inconsistently. A model may succeed on a task in one run and fail the next. Single-score reporting does not characterize that variability.
Developers can train models, intentionally or inadvertently, on benchmark data, inflating performance scores without reflecting actual ability. This “data contamination” arises from memorization of training data, and most developers do not track or disclose it. Competitive evaluation regimes create a related incentive to overfit benchmark tasks for leaderboard position rather than assessing the underlying capability represented by the benchmark.
Models now regularly identify when a prompt serves an evaluation purpose and can change normal outputs to score well; some have exploited loopholes or responded in ways that developers did not anticipate, like in the U.K. AI Security Institute cheating findings.
Benchmark quality is also hard for researchers to determine given nontransparent methods or invalidated metrics. Several metrics in current benchmarks, such as “memorization” and “harmful content,” have abstract and inconsistent definitions across evaluations.
Evaluation design rarely reflects affected communities
Defining a risk or capability is a subjective value determination that shapes how evaluations are scoped before technical decisions are made. Communities most likely to be affected by foundation model deployment are rarely meaningfully involved in evaluation design or in conducting the evaluations that assess those harms.
Evaluators who do not represent the affected communities may not understand the full extent of real-world impacts. Many evaluations developed in the United States may not measure different cultural contexts that have different norms for sensitive content, ethics, and harm. Few evaluation suites are built or validated outside where they were designed. Even when community participants are engaged, they are often uncompensated and lack meaningful ability to change or halt development. More evaluations cover safety-related harms than ethical and social risks. A model that refuses harmful requests 98% of the time may fail disproportionately in one language, for one dialect, or against one group of users.
Who should do the evaluations, and under what conditions?
Access and independence are in tension
First-party evaluators are more likely to have access to proprietary and predeployment systems. Third-party evaluators face challenges in lack of access to datasets and models, which constrains how they can test. The rapid development pace of GPAI puts pressures on developers to keep important information proprietary and prioritize speed over risk management. Independent auditors tend to conduct higher quality evaluations, but how to measure independence and the quality of evaluations remain open questions.
Access itself is a substantial challenge: Meaningful safety testing often requires evaluators to bypass or disable safety guardrails, risking that those mechanisms are weakened or replicated outside the controlled setting and potentially causing safety incidents.
Companies can compromise the integrity of evaluation regimes by selectively disclosing audit-relevant information, supporting biased research, gaming outcomes, and shaping the criteria, metrics, evaluator requirements, and institutions involved.
Organizational capacity lags behind evaluation ambitions
A 2026 review by researchers at Microsoft Research, Carnegie Mellon, and UCLA, covering 161 empirical studies by industry practitioners, found more than 120 report that practitioners tasked with responsible AI work lack the expertise and training in relevant technical and sociotechnical domains to carry it out. In fairness testing, several studies found practitioners unable to distinguish between the metrics they were applying. Practitioners described teams as understaffed and roles as fragile; and multiple studies concluded that evaluation processes could become isolated box-checking exercises. Rigorous evaluations require resourcing time, money, and skilled labor. Even a well-designed evaluation policy relies on the organizational capacity to run it. The preprint covers literature from 2018 through August 2025 and is U.S.- and U.K.-weighted.
What should happen to evaluation results?
Reporting guidance outpaces reporting practice
Section 3 of the NIST AI 800-2 initial public draft describes statistical analysis and uncertainty quantification (3.1), sharing evaluation details and data (3.2), and reporting qualified claims (3.3). Section 3.1 notes that benchmark results are commonly reported as single numbers or percentages, without sample sizes, variance across runs, or confidence intervals; Section 3.3 emphasizes that results must be read within the conditions of the test and not as a general property of the model.
The Center for Democracy and Technology commended CAISI on the draft’s clear measurement definitions, and the AI Standards Lab suggested that benchmark popularity should not be used as a proxy for validity. Both argued that the draft should treat evaluations as iterative, with the AI Standards Lab commenting that the draft treats benchmarks as static when they degrade through data contamination and semantic drift.
Several jurisdictions have documents detailing how to report evaluations, including China, the EU, Singapore, and New York, as do the Bletchley Declaration, the International AI Safety Report, METR’s Key Components of an RSP, NIST’s AI Risk Management Framework, the U.K. AISI’s AI Safety Institute Approach to Evaluations, and company policies at Anthropic, Google DeepMind, Meta, Microsoft, OpenAI, and xAI.
For example, California’s S.B. 53 requires developers to publish public transparency reports for large frontier models and submit risk summaries to state authorities, while the EU AI Act routes serious incident reporting to the AI Office and national market surveillance authorities rather than to the public. New York City’s Local Law 144 requires an annual independent bias audit to be posted publicly, although student investigators were not likely to find these reports given the findings of the 2024 study.
Model and system cards are a widely adopted documentation format, proposed by Mitchell et al. in 2018 as short, fixed sections covering intended use, performance across relevant groups, and limitations. They have been voluntary so far, but the EU AI Act requires GPAI providers to supply more structured information to downstream providers and to publish a training content summary according to an AI Office template. Those obligations applied starting on Aug. 2, 2025, and the commission’s enforcement powers began Aug. 2, 2026.
Evaluation results do not always reach decisionmakers
Reliable evaluations often do not reach decisionmakers and stakeholders. An evaluation will be more likely to have impact if embedded into existing governance structures, shared with decisionmakers, and conducted before decisions are made. Many evaluations are first party and designed for internal use, or academic and theoretical, and thus are less helpful for policymaker needs and less likely to translate to real-world risks. Few incentives exist to build evaluations oriented toward public rather than commercial interests.
Even when results reach decisionmakers, translating them into action is not simple. Officials weighing a procurement decision may lack the technical expertise to interpret a benchmark score or red-teaming result, and often face competing priorities—speed, cost, or competitiveness—that cut against acting on results. Alongside wider development of evaluation subjects and methods, there is a need for systems of transparency and accountability for the results.
On the sidelines of the 2026 Paris G7 Digital Ministerial, Robust Open Online Safety Tools (ROOST) and Mila signed a partnership to develop and distribute open-source safety tools, collaborating on “research, evaluation, datasets, and tool development.” The initial focus is youth safety and mental resilience around AI companions, with organizers intending to expand efforts. Open-weight and open-source tools could provide more transparency and adoption than internal proprietary methods, but confidential evaluations may decrease risks of evaluation gaming. The trade-off remains an open question.
Procurement is one lever through which evaluations lead to action. Procurement offices may make decisions based on published results. The U.S. General Services Administration’s draft contract terms address AI-related procurement in a new proposed general services acquisition regulation. The proposed clause, since revised, addresses six areas of concern for the federal government in procuring AI systems: intellectual property and data rights; data handling and processing; contractor compliance, reporting, and documentation obligations; privacy protections; data portability and interoperability; and AI performance and evaluation standards.
Evaluations can also drive company behavior. Given certain results, a developer may adjust model design or training, add safeguards and mitigations, delay model deployment until thresholds are met, or restrict model or weight access. Most companies, however, are not obliged to act on evaluation results and do not consistently take these steps.
Evaluations are often one part of a larger multi-layered “defense in depth” strategy that can include audits for legal compliance, disclosure requirements, and post-market monitoring. The Swiss Cheese Model, used in risk analysis, illustrates defense in depth as several layers of Swiss cheese with holes: A defense layer may have gaps, but as long as another covers them, the defense holds. Model evaluations are one valuable layer for assessing capabilities and risks, but many experts agree they should not be singularly relied on without complementary evidence from other sources.
- What claims does an evaluation result support?
- Measured capability depends on prompting, scaffolding, and resources, so incomplete elicitation yields a lower bound of unknown tightness. What may a policymaker legitimately infer from a model that passes, and what are passing results currently used to support?
- Models now recognize evaluation conditions, and documented cases show them cheating and underreporting it. What does that do to the evidentiary standing of automated benchmarks, and what would restore it? How are evaluations themselves to be validated?
- The unit of evaluation is shifting from the model to the deployed system, which can include model plus tools, permissions, memory, and users. Whose mandate covers that or whose should it be?
- Who should evaluate what, and what does independence require?
- Which evaluations can only be performed by a particular class of evaluator, and which are currently being done by a given class because of access rather than fitness?
- What are the implications of a commercial evaluation market, and what parallels exist with other assurance markets? What does independence mean here, and how can it be assessed?
- Government evaluators obtain predeployment access through voluntary agreements and memoranda of understanding. Is conditional access compatible with independence? If not, what is the alternative? Statutory access, safe harbor, mandated third-party review? And who bears its costs?
- Affected communities are rarely engaged in evaluation design and are often uncompensated and unable to change or halt deployment. Does meaningful participation require compensation, agenda-setting power, or the ability to stop something? Who should pay for it?
- How should evaluators consider convergence, customization, and compatibility of results?
- Convergence on terminology, reporting formats, methods, or thresholds all carry very different costs and politics. Where is alignment worth pursuing, and where is divergence the right answer?
- How can networks like NAAIMES best build interoperability between predeployment evaluations and cross-border capability testing? What are the costs, benefits, and challenges?
- NIST AI 200-2 builds assessments around an organization’s own TEVV objectives rather than a fixed test set. Customized assessment buys fitness at the cost of comparability. Who needs comparability most, and what do they lose?
- Open-source tooling lowers the cost of entry and improves transparency, but published methods are also easier to game and to train against. Where should the line fall, and does the answer differ for capability benchmarks versus safety evaluations?
- How should transparency and accountability be addressed?
- Evaluation results now trigger export controls, deployment restrictions, and procurement decisions. What process should attach to a consequential determination based on classified evaluation results, for the developer, and for the public?
- What can be disclosed publicly, what on a restricted basis, and what must remain secret? Who decides, on what criteria, and is that allocation of authority right?
- Most developers are not obliged to act on their own evaluation results and do not consistently do so. What incentives or requirements (e.g., procurement conditions, liability, disclosure requirements) could change this? What findings should trigger what actions?
The dialogue convened days after the publication of a widely read essay from a frontier AI lab leader calling for independent third-party evaluators embedded in frontier AI companies, and the postponement of OpenAI’s initial public offering attributed to safety concerns. Participants agreed that independent, third-party evaluation is now necessary but offered different perspectives on what to measure, where in the AI stack to measure it, who should be trusted to do it, and what should follow from the results.
Evaluation obligations are now enforceable in the European Union under the AI Act, but regulators are still building enforcement capacity. EU regulators described evaluation as central to managing systemic risk. Obligations are operationalized through a voluntary code most leading developers have signed, with direct enforcement powers in effect since Aug. 2. Providers must conduct internal model evaluations using standardized protocols and “state-of-the-art” standards and give external evaluators access to assess models. Regulators reported progress toward embedded evaluators with greater internal access, a forthcoming blueprint for structured pre-release model access, and public funding to build evaluation capacity. Finding external evaluators who can build trusted relationships with Chinese providers and assess their models remains a challenge.
Participants disagreed over whether benchmarks are necessary and warned that they are increasingly unreliable because of overfitting, training data contamination, a skew toward capability over safety and costs, and weak correlation to real-world tasks. The problem may be structural. As one participant stated, once the benchmark becomes the objective, the benchmark measures nothing. Some agentic systems appear to recognize when they are being evaluated and behave differently. Chain-of-thought legibility, which has underwritten much practical auditing, may erode as developers build models that reason less in natural language. Several participants argued that evaluation capacity must include pre-deployment testing, internal audits, and post-deployment monitoring.
Model-level benchmarks tell deployers little about behavior in production. System-level benchmarks, that include guardrails, harnesses, and business logic, can provide more meaningful insight. One participant extended the unit of analysis further to the human–AI pair. Testing AI applications requires analyzing sectoral use cases, rather than applying cross-cutting benchmarks. Participants also urged measuring upside, not only risk: We ask whether aircraft are safe, but also whether they are efficient; without quantified benefits, non-catastrophic trade-offs cannot be weighed in the full context.
Granting external evaluators internal access leaves open questions of independence, as funding arrangements remain opaque, even when evaluators take no direct money from labs. One way to increase trust is to publicize the terms between evaluators and developers. Standards development can establish how to frame evaluations: testing and evaluation are methods, verification and validation are purposes, and an evaluation must be targeted at the specific claim it is meant to support. One industry representative reportedly said a prominent benchmark was built for “marketing,” which may make it unsuitable for procurement or fitness-for-use decisions.
There is a mismatch between evaluation capacity, which is advancing quickly, and enforcement infrastructure (reporting, liability, and insurance), which is still being debated in many jurisdictions. Evaluations produce evidence, but companies face disincentives to involve law enforcement, and many agencies best positioned to respond lack the authority. Participants characterized the courts as moving toward a product-liability framework and away from fault, toward something closer to strict liability, creating stronger incentives than current voluntary commitments. Insurers want to underwrite AI risk but cannot, absent standards for how models are evaluated and matched.
Two constituencies are underrepresented in current discussions: the national security community, which in some cases is building an insular parallel evaluation ecosystem despite similar underlying methodologies, and the public affected by a diverse set of non-catastrophic harms, including discrimination, privacy violations, and threats to physical safety, which are more frequent and have received less media and policymaker attention. Proposals that address only catastrophic risk, one participant argued, are unlikely to achieve legitimacy.
The Brookings Institution is committed to quality, independence, and impact.
We are supported by a diverse array of funders. In line with our values and policies, each Brookings publication represents the sole views of its author(s).
Leave a Reply