What Are AI Evals? Testing Intent, Not Just Output

An agent picks up a ticket. It writes the feature, writes the tests, and the suite goes green. The code does exactly what the code says. It also solved the wrong problem, because it misread one line of the requirement.

Nothing in that pipeline can catch it. The asserts came from the same misreading as the implementation. They agree with each other, and both are wrong.

That gap is what evals exist to close.

What an AI eval actually is

An assert compares a value to a fixed expectation and returns true or false. Status code 200. This string. That return value. Binary, exact, and blind to meaning.

An eval scores an answer against a rubric, a reference, or a set of properties the answer must hold. It passes on a threshold instead of an exact match. That one change reaches something an assert cannot: whether the output means the right thing.

Acceptance criteria are written in sentences. The refund policy must be stated in plain terms. The summary must not invent a figure. Until now those sentences were validated by a person reading output, or not at all, because no assert could hold them.

An eval can. It turns an acceptance criterion into something a pipeline runs on every commit.

Why asserts alone fail when a model writes the code

Traditional testing rests on an assumption that is now shaky. It assumes the person writing the test understood the requirement, and that a failing test means the code is wrong.

Hand the work to a model and both halves weaken. The model writes the code and often the tests too. If it misread the requirement, it encodes that misreading twice. The suite passes. The build ships. The requirement is still unmet, and nothing in CI has an opinion about it.

Asserts check the code against itself. Evals check the output against what a person asked for.

There is a second effect worth naming. A written eval is a requirement the model can read. Feed the acceptance criteria into the loop as scored checks and the target stops being “make the tests pass.” It becomes “satisfy the stated intent.” The eval keeps the requirement present in the work instead of lost after the first prompt.

Evals are necessary and insufficient

Here is the failure evals exist to catch and routinely miss.

An eval scores an answer at 0.94 similarity against the reference. The answer flipped the sign on a financial figure. The metric was happy. The output was wrong.

Both halves matter. Evals are necessary, because nothing else catches drift in a system whose output shifts run to run. They are also insufficient in a way you can predict. They measure how much an output resembles an expectation, and resemblance is not correctness.

Much of an AI feature is fixed code. Prompt assembly, retrieval, tool dispatch, JSON parsing. That surface is ordinary software and deserves ordinary tests. Evals are the right tool only for the part that generates language. Teams that route everything through a judge end up paying a model to confirm that JSON parsed. The split between those two surfaces is worked through in how to test AI systems, the practical companion to this piece.

Where AI evals earn their keep

Evals are dependable when the answer is checkable and the output space is narrow.

• Classification. Intent routing, spam or not, was this refusal right. Small answer space, cheap labels, easy to audit when judge and human disagree.
• Extraction and structure. Did it return the seven fields, in schema, with entity counts matching the source. These hold or they do not.
• Rubric-scored summaries. Does the summary carry the three facts on the checklist, stay under the length cap, and add no entity the source lacks.
• Refusal and safety limits. Regression here is the whole reason to run a suite. The failure is silent and the blast radius is public.
• Coarse sentiment. Broad buckets work. Fine gradations do not.

One pattern runs through all five. The rubric fits in a sentence, and two competent people reading the same output would score it the same. Break that condition and the score stops measuring the property you named. It starts measuring the judge.

Where LLM-as-judge breaks

Most suites past the classification stage use a model to grade a model. It works better than critics claim and worse than the dashboard suggests.

The founding study, Zheng et al. on MT-Bench and Chatbot Arena, found a strong judge agreed with human preference over 80% of the time. That is about the rate humans agree with each other. The same paper named the failure modes, and they have proved durable.

Position bias. Shown two answers, judges prefer the first at rates well above chance. Fix it mechanically: score both orderings, keep the verdict only if it survives the swap. Grade one ordering and you are measuring page order.

Verbosity bias. Longer answers score higher whatever they contain. If your rubric does not punish padding, your eval rewards it, and you will promote the chattier model.

Self-preference. Judges favor text from themselves or their own model family. Grade with a sibling and you build a loop that ratifies its own style.

The harder limit is domain reasoning. A judge handles “is this a summary” well. It handles “is this arithmetic right, given the twelve constraints in the source” badly. Hallucination over domain logic, subtle contradiction, sycophancy, and format preference all return a confident number with no tie to truth.

False confidence follows. Surface metrics stay high while factual drift builds underneath, because nothing is measuring facts. Embedding scores share the gap. Metrics like BERTScore measure semantic similarity, and similarity is blind to the things that decide correctness. Magnitude, sign, and polarity all move the answer without moving the score. Studies in specialist domains report the obvious result: near-identical high scores across outputs of clearly different quality.

Which brings back the opening. The flipped figure scored 0.94 because a sign barely moves a similarity metric and completely inverts the answer.

The number moves when nothing changed

Most teams assume temperature zero makes a suite repeatable. It does not. Learn why before you lose a week chasing a two-point drop with no cause.

Thinking Machines Lab documented the mechanism in September 2025. They sampled 1,000 completions from Qwen3-235B at temperature zero on one fixed prompt. They got 80 distinct outputs. Every run agreed for 102 tokens, then split at token 103. Nothing about the request changed.

The cause is not sampling. It is not simply floating-point order either, which is the usual explanation. Inference servers batch requests as they arrive, and the reduction kernels are not batch-invariant. The shape of the reduction depends on how many other requests were in flight. Your numbers depend on other people’s traffic. The same work showed it is fixable with batch-invariant kernels, at a real throughput cost, which is why hosted endpoints do not do it by default.

Three consequences for a suite in CI:

1. One run is a sample, not a measurement. Re-run the same suite on the same build and the number moves. Find that noise floor before you read any delta.
2. Small suites cannot see small regressions. Eighty questions with a three-point noise band will never surface a two-point regression.
3. Bit-exact runs are a debugging tool, not a CI mode. Reach for deterministic kernels to isolate one failure. Do not pay for them on every run.

Put error bars on the score

The fix is not more discipline. It is statistics, and there is a reference for it.

Adding Error Bars to Evals, by Evan Miller at Anthropic in November 2024, argues that an eval is an experiment and should be read like one. A score is a mean over a sample of questions drawn from the much larger set you could have asked. A mean with no standard error is not a result.

Two findings change how you build a suite.

• Report standard errors next to every mean. The question is rarely “did the score drop.” It is “did it drop by more than the interval.” Without the interval you cannot answer, and you will ship on noise and block on noise about equally often.
• Clustered questions break the independence assumption. Many suites fan out several questions per source document. Those are not independent samples. The paper reports clustered standard errors exceeding naive ones by more than 3x. Ten variations per seed document buys far less power than the question count suggests.

In practice, track question count, cluster structure, and the interval beside the headline number. A suite reporting “87%” and one reporting “87% plus or minus 4, n=200, 40 clusters” cost the same to run and differ completely in what they let you decide.

This is the discipline the test coverage conversation needed and mostly missed. A number with no error term invites the behavior you least want, which is optimizing the number.

Building an oracle when there is no golden answer

Underneath all of this sits the oracle problem. Verification needs a known correct answer, and open-ended output often has none. Three strategies give you a usable oracle anyway. None needs a reference output.

Structural invariants

Assert the properties the answer must hold whatever its wording. Schema. Entity counts matching the source. Citations that resolve. Numbers inside a plausible range. Required facts present, forbidden facts absent.

Invariants are cheap, exact, and catch what similarity misses by design. A flipped sign trips a sign check even at 0.94 similarity. Start here. This layer needs no judge and adds no noise.

Metamorphic testing

When you cannot state the right output, you can often state how it must change when the input changes. Metamorphic tests assert relations between runs instead of absolute values, which steps around the oracle problem. The idea predates LLMs. CheckList (Ribeiro et al., 2020) formalized it for NLP with invariance tests, where a label-preserving edit must not move the prediction, and directional tests, where it must move a stated way.

Applied to a feature: paraphrase the question and the answer should hold. Add an irrelevant sentence and the extraction should not shift. Double a quantity and the total should double. Reorder retrieved documents and the conclusion should stand. Every one of those is testable without deciding the right answer.

Consensus and ensemble voting

Run several models, or several prompting paths, over the same input and compare. Tight clustering across independent systems is weak evidence of correctness. A lone outlier is strong evidence of a problem. The oracle comes from the median behavior of the group.

The caveat matters more than the technique. Models from one family share training data and failure modes, so siblings agree confidently and wrongly. Consensus buys you something only when the members are genuinely independent.

None of this is specific to language models. It is the standard toolkit for scientific and cluster workloads, where nondeterminism killed exact-match assertions decades ago. HPC testing hit this wall first, and the vocabulary is worth borrowing whole.

Traceability is an engineering requirement

Linking every result to the prompt, model version, index, and dataset that produced it is not a governance checkbox. It is the only way to answer the question you will be asked, which is “what changed.”

When a score drops, the candidates are a prompt edit, a provider-side model update, an index rebuild, a parameter change, or plain noise. With no provenance you cannot tell them apart, and you will spend the afternoon guessing. Providers ship updates on their own schedule, so “nothing changed on our side” is a claim your logs have to support.

The tooling has settled enough to adopt rather than invent. The OpenTelemetry GenAI conventions define span attributes for this, including gen_ai.request.model for what you asked for and gen_ai.response.model for what you got. That pair settles most silent-upgrade arguments. Standard spans also mean your eval results, production traces, and incident timeline share one vocabulary.

At minimum, record per run: prompt template version, the resolved model from the response rather than the request, index version, sampling parameters, dataset revision, and the harness commit.

Where evals fit in the pipeline

Tooling comes last on purpose. The layering matters more than the framework.

The open-source options cluster by strength. Ragas focuses on retrieval and grounding for RAG. DeepEval targets component and agent metrics with CI hooks. Promptfoo compares prompts and models side by side and ships a red-team suite. Lighter harnesses such as OpenAI Evals and Inspect give you reusable templates, model-graded scoring included, for building task tests fast.

Pick by what you are testing, then layer the suite so it fails fast:

1. Deterministic tests on the non-generative surface. Templating, parsing, tool dispatch, retrieval plumbing. Milliseconds, no model calls, no flake.
2. Structural invariants on generated output. Schema, counts, ranges, citations. Fast, exact, high signal.
3. Metamorphic relations across paired runs. Slower, still no reference answer needed.
4. Judge-scored rubrics on the subset where a rubric truly applies, with order swapping and reported error bars.
5. Adversarial and injection suites as their own gate. Prompt injection is still LLM01 in the OWASP Top 10 for LLM Applications, holding first place across editions, and a system prompt edit is exactly what reopens the hole.

Layers one to three run on every commit. Layer four runs on every prompt or model change, which is rarer than every commit and more often than teams assume. Layer five is a release gate.

The three things that matter most

• An eval validates intent. An assert validates behavior. You need both. When a model writes the code, the assert can agree with the bug, so the eval is the only check still reading the requirement.
• A score with no error bar is not a measurement. Find the noise floor by re-running the same suite on the same build. Report the interval, the count, and the cluster structure every time.
• Provenance separates a diagnosis from a guess. Log the resolved model, prompt version, index version, and dataset revision on every run.

The instinct that transfers best from QA is knowing what a test actually asserts. It is worth more here than almost anywhere else, because eval suites are very good at producing an authoritative number about nothing in particular. The role that owns this work is covered in What Is an SDET in 2026?, and the shift that created it in Testing in the Age of AI.

Related reading

• How to Test AI Systems. The practical companion: what to assert exactly, what to sample, and how to gate CI on a pass rate.
• HPC Testing. The same no-oracle problem, solved decades earlier on clusters.
• Testing in the Age of AI. Why testing moved back inside the development loop.
• Using AI Agents for Testing. The other direction, models as the tester rather than the thing tested.
• Test Coverage Explained. Why a number with no error term invites gaming.