Understanding Evals

An eval is a repeatable way to measure how well an AI system performs the job you actually built it to do. That sounds simple, but it represents one of the biggest shifts in how AI products are built.

With traditional software, much of quality can be specified explicitly. Given this input, return this output. If a user clicks this button, this action happens. A unit test tells you whether the behavior is correct. AI systems are different. There may be many acceptable answers to the same question. The model may behave differently across runs. A response can be technically correct but unhelpful. An agent can produce a convincing final message while taking the wrong action in the background. And a model that performs exceptionally well on public benchmarks can still be the wrong model for your users.

So instead of asking only “Does the system work?” AI teams increasingly need to ask: how well does it perform the job, where does it fail, how reliably does it behave, and is this version actually better than the one before it? That is what evals are designed to answer.

One Case Tells You Little; a Set Tells You Everything

Imagine building an AI support assistant. A single eval case: a customer says they were charged twice and asks for a refund. The expected behavior is that the assistant identifies the billing issue, uses the correct refund policy, avoids promising something it cannot do, escalates when required, and responds appropriately to the customer's frustration. From that one case you can measure several things independently: was the policy correct, was the response grounded in available information, did it choose the correct escalation path, did it complete the required action, was the tone appropriate, how long did it take, and how much did it cost?

One example tells you very little. But build a set of representative cases covering normal usage, difficult situations, important customer segments, and known failure modes, and you have something far more powerful: a measurement system for the behavior of your product. Now you can run the same tasks against a new prompt, a different retrieval strategy, another agent architecture, or a new model.

Instead of “this prompt feels better,” you can say “task completion improved from 76% to 84%, but escalation accuracy dropped.” Instead of “the new model looks smarter,” you can say “the new model performs significantly better on complex research tasks, but our current model remains faster, cheaper, and more reliable for the workflow that represents most of our usage.” And instead of discovering a regression from customers after release, you can often catch it before the change reaches them.

This is where evals become much more than testing. They become the feedback loop through which an AI product improves: build, evaluate, inspect failures, improve, evaluate again, ship, observe production, add new failures to the eval set. Over time, the eval system becomes a record of what the organization has learned about the product. It captures what good means, it remembers what has gone wrong before, and it gives the team a way to compare future changes against that accumulated knowledge. That is why evals sit somewhere between a test suite, a product specification, and a measurement system, and why they are increasingly a product-management responsibility.

Why This Matters: Most AI Products Still Ship on Vibes

Despite how important evaluation has become, a surprising amount of AI development still looks like this: you tweak a prompt, run a handful of hand-picked examples, the demo looks solid in the war room, and the team moves on. Then production hits. Users edit the outputs more than they accept them. Support tickets cluster around failure modes you never tested. A new model wins the public benchmarks but quietly regresses on your highest-value workflow. And when the CEO asks why output volume doubled while satisfaction fell, you have no clean answer.

Generative systems make traditional QA insufficient, not obsolete. You should still unit-test everything deterministic: schemas, permissions, business rules, tool arguments, calculations, APIs, and state changes. But you cannot write one deterministic test that proves a model will always stay on-brand, remain faithful to a messy source document, handle ambiguity appropriately, or recover gracefully when a tool returns unexpected information. Accuracy alone isn't enough either: a system can be factually correct and still fail because it's too slow, too expensive, unhelpful, poorly calibrated, unable to complete the task, or unpredictable when it matters.

Product quality in AI lives at the intersection of the Trinity. Evals are how you make that intersection measurable.

Why Evals Are Now a Core PM Skill

Andrew Ng has argued for years that evaluation is one of the biggest bottlenecks in building useful generative AI applications, particularly when outputs are open-ended and there's no single obviously correct answer. The underlying idea is simple: if you can't measure whether a change improved the system, iteration becomes guesswork.

Public benchmarks are still useful. Arena, SWE-bench, Terminal-Bench, MMLU-style academic benchmarks, coding evaluations, and provider model cards tell you something about the general capabilities of a model. But they answer “how capable is this model in general?” Your product needs to answer “how capable is this system at our specific job?” Those are different questions. Your support agent may need to recognize when a frustrated customer should be escalated. Your contract-review system may need to identify prohibited clauses without inventing legal language. Your research agent may need to search multiple sources, preserve citations, distinguish evidence from inference, and admit when information is missing. A generic reasoning benchmark can't tell you whether the system does those things well. That requires a benchmark built around your workflow.

Aaron Levie has made a similar argument from the enterprise side: general model benchmarks show the direction of AI progress, but the larger opportunity is evaluating the real workflows companies perform, eventually down to the specifics of a company's data, policies, and processes. The implication is straightforward: you cannot reliably automate work you cannot assess.

Claire Vo's evolving How I AI Bench offers a practitioner version of the same idea. Instead of asking which model is universally “best,” she built repeatable tests around work she actually performs: writing PRDs, critiquing prototypes, generating wireframes, solving technical tasks, and assessing the personality and usefulness of an assistant. The interesting result wasn't simply which model ranked first. It was that the automated judges and her own preferences sometimes disagreed. The model judges rewarded constraint-following and technical completeness; her human review cared more about usability, originality, clarity, aesthetics, and whether she'd actually want to use the result. That disagreement is useful. It exposes a gap in what the evaluation is measuring, which is exactly what a product-specific eval should help you discover.

From the Field: Workflow-Specific Evaluation

Box provides a strong example of workflow-specific evaluation. Its Complex Work Eval doesn't ask models isolated trivia questions. Instead, the model operates inside an agent workflow and has to perform end-to-end knowledge work across realistic documents: retrieving information, reading PDFs, analyzing spreadsheets, processing presentations, interpreting images, combining information across multiple files, and producing a useful deliverable. The benchmark spans a range of industries and task types: data analysis, due diligence, report drafting, document review. The important part is that the model is tested on the work, not just on a prompt.

The results also show why PMs should care about segment-level performance. In Box's evaluation of a newer model against the prior generation, the overall improvement was relatively small. At first glance that might suggest the newer model is only marginally better. But the breakdown told a very different story: on data-analysis tasks the improvement was much larger, and in some industry-specific subsets the gap widened dramatically again. Same two models. Completely different product decision depending on the workflow. That is the difference between asking “which model is better?” and “which system is better for this job, for these users, under these constraints?” The second question is the one a PM needs to answer.

Designing Evals That Measure What Matters

The hardest part of evaluation isn't running the benchmark. It's deciding what belongs in it. A good eval starts with the job your product is supposed to accomplish and the ways that job can fail.

Start With Failure Modes, Not a Metric Wishlist

A common mistake is to gather the team and brainstorm what “AI quality” means. Accurate. Helpful. Safe. Fast. Friendly. Those words sound sensible but are too abstract to evaluate well. Instead, start with evidence. Pull production traces. Read conversations. Review thumbs-downs. Look at abandoned tasks. Watch how users edit outputs. Read support tickets. Sit with domain experts and ask why specific outputs are unacceptable. Then build a failure taxonomy.

For a support agent

• invented policy details
• wrong escalation decisions
• failure to use previous conversation context
• correct answer but inappropriate tone
• unnecessary tool loops
• failure to complete the requested action

For a RAG product

• wrong document retrieved
• correct document retrieved but the relevant passage missed
• unsupported factual claims
• answer contradicts the source
• relevant evidence omitted
• correct facts but incomplete synthesis

For an action-taking agent

• wrong tool selected
• correct tool with wrong arguments
• duplicate actions or unnecessary repeated calls
• wrong record modified
• required confirmation skipped
• action completed but user not informed
• failure to recover from a tool error

This is why tracing matters. Was the wrong document retrieved? Did the model misunderstand the evidence? Did a tool return bad data? Did the orchestration layer make the wrong decision? Did the final response introduce an unsupported claim even though the correct information was available? The final answer is only the visible endpoint of the system. Observe the failure first. Instrument the eval second.

Building the Eval Dataset

Once you know the job and the major failure modes, you need examples. There's no universal number that makes an evaluation “real.” Early in development, 20 to 50 high-signal cases can already provide enormous value. If a change creates a 30-point improvement, you don't need thousands of examples to notice it. As the product matures, improvements get smaller and subtle regressions get costlier, so the set should grow. A useful eval set covers several categories:

Purely random sampling from production can miss the high-risk cases because they're intentionally rare. That's why eval sets should often be both distribution-aware and risk-weighted.

Real Data Beats Imagined Data

Synthetic examples are useful: fast to generate, good for coverage, especially before a feature launches. But synthetic data has a dangerous property: it reflects the assumptions of the people who created it. Real users don't. They misspell things, omit information, contradict themselves, ask for two things in one sentence, use internal jargon, give irrelevant context, and behave in ways nobody on the product team anticipated. As soon as production data is available, real traces should become one of the primary sources of eval cases. The strongest suites usually combine real production traces, high-fidelity simulations, explicitly designed edge cases, adversarial tests, and historical failures.

Don't Make One Dataset Do Everything

A mature evaluation system usually benefits from separating capability evals and regression evals.

Capability evals

These ask how good can the system become? They should contain difficult tasks the current product doesn't always solve. If your model scores 100% on every example, the benchmark may no longer tell you much about future improvements. Capability evals are useful for exploring better models, improved prompts, retrieval strategies, tool designs, agent architectures, reasoning configurations, and new product capabilities. You want room to improve.

Regression evals

These ask did we break something we already know how to do? They should mostly pass. Once a problem is solved, the corresponding case becomes permanent regression coverage. A model once invented refund-policy exceptions; you fix it; the failing trace becomes a regression case; six months later a new model or prompt change causes the failure again, and the eval catches it before users do. As the product improves, examples effectively graduate from the capability suite into the regression suite. You need both: a stable suite that protects what already works, and a challenging suite that shows what still needs improvement.

Keep Some Cases Hidden

Once the eval becomes important, teams start optimizing against it. They run the same 100 cases repeatedly, inspect every failure, tweak prompts specifically to fix those examples. Eventually the score rises and production doesn't. You've started teaching to the test. This is why some cases should be held out from everyday optimization. Your team can work against a development set while a separate held-out set gives a cleaner estimate of whether the improvement generalizes. The goal is not to maximize the benchmark score. The goal is to estimate real-world performance.

What Should You Measure?

Once you have the tasks, you need to define success, and you shouldn't collapse everything into one number too early. A useful way to think about measurement is through four categories: capability metrics, guardrail metrics, operational metrics, and product/business metrics.

1. Capability metrics

These measure whether the AI does the job well: task success, factual correctness, faithfulness, groundedness, completeness, answer relevance, retrieval quality, tool-selection accuracy, escalation accuracy, quality of synthesis, recovery from ambiguity, user usefulness. The most important is often task success: did the system actually accomplish the user's goal? A travel agent that gives an elegant explanation of flight options but fails to book the flight has not succeeded. A support agent that writes a slightly awkward response but correctly resolves the billing issue and updates the account may have. The product's job should define the metric.

2. Guardrail metrics

Some dimensions aren't things you gradually optimize. They're constraints: unsupported factual claims, hallucinations in high-stakes contexts, privacy violations, policy violations, safety violations, incorrect irreversible actions, unauthorized tool use, fabricated citations, broken format contracts. You might accept a 2-point loss in writing style for a 20% cost reduction. You probably don't accept a new model that occasionally sends money to the wrong bank account. Guardrails should be treated as release gates, not averaged into an overall quality score.

3. Operational metrics

AI quality has a physical cost: latency, p95 latency, cost per request, cost per successful task, number of model calls, tool calls, retries, token usage, timeout rate, failed tool calls. These often change the decision. Consider Model A at 88% task success, $0.04/task, 2.8s p95 versus Model B at 91%, $0.19/task, 8.2s p95. Is B better? For legal due diligence, the extra quality may be worth it. For autocomplete inside a messaging app, eight seconds makes the feature unusable. Quality doesn't exist independently of the user's experience or the business model.

4. Product and business metrics

Offline quality isn't the goal. User value is. Eventually you connect eval performance to what happens in the product: AI-suggestion acceptance rate, edit rate, support deflection, task completion, time-to-resolution, time saved, conversion, retention, repeat usage, user satisfaction, escalation rate, revenue, cost savings. Suppose your offline helpfulness score rises 12% but users still rewrite 40% of responses. Something's wrong: maybe your grader rewards verbosity while users want concision, maybe the dataset doesn't represent production, maybe the real problem is latency or trust. An offline score that improves while the product metric doesn't isn't necessarily a failure. It's information: your current measurement system doesn't fully capture user value. That's how the eval itself improves.

Two Evaluation Loops: Offline and Online

Offline evals run in a controlled environment before promotion. You run the same tasks against a new model, a different prompt, a retrieval change, a new tool, a new agent architecture, or a different context strategy. They answer: did this change improve the system, which dimensions improved, which regressed, is the improvement repeatable, is it safe to ship, is it worth the cost? This is the development and release loop.

Online evals run against, or sample from, real production interactions. They answer: are new failure modes appearing, has quality changed since release, are particular customer segments struggling, has the input distribution changed, are users behaving differently than expected, is the offline benchmark still predictive of real product quality? The two loops should feed each other. Your offline suite should slowly become a compressed representation of what you've learned from production.

Three Ways to Grade an Eval

Once you know what to measure, you need a grader. There are three families: programmatic, model, and human. Strong systems combine them.

1. Programmatic / code-based checks

Whenever expected behavior is deterministic, use code: valid JSON, schema compliance, required fields, exact tool name, calculation correctness, correct record ID, state successfully changed, required citation present, maximum tool calls, latency below threshold, permission rule respected. These are fast, cheap, and reproducible. Don't ask an LLM “does this JSON seem valid?” when a parser answers perfectly. Don't ask an LLM whether the CRM record changed correctly if you can query the actual record. For agentic products this is especially important: the final answer may say “Done. I updated the customer's address,” but if the system changed the wrong record, response quality is irrelevant. Check the state of the world, not only what the AI says happened.

2. LLM-as-a-judge

Some dimensions can't be reduced to deterministic logic: helpfulness, tone, faithfulness, completeness, ambiguity handling, quality of a synthesis, whether an answer is overconfident. Here another model can grade, but the key is a clear rubric. A weak prompt: “Rate this answer 1 to 10.” A stronger one: “Determine whether every material factual claim is supported by the provided sources. Return PASS, FAIL, or UNKNOWN. For FAIL, identify each unsupported claim.” Model judges perform better when asked to classify, choose between alternatives, check against explicit criteria, compare two outputs, or identify violations, and worse when asked for vague global quality judgments.

A model judge shouldn't be trusted just because it produces a score. Create a human-labeled set, preferring domain experts where expertise matters, then compare the judge against those labels and inspect false positives, false negatives, disagreements, and ambiguous cases. Improve the rubric, add examples, clarify definitions, run again. There's no universal “90% agreement is good” rule; the required reliability depends on the task. A marketing-tone grader may tolerate more disagreement than a judge detecting unsupported legal claims. And be careful with simple agreement percentages: if 95% of examples are obvious passes, a judge can look highly accurate while missing the rare failures you actually care about. Ask “when this judge says the system is safe to ship, how often is it wrong?”

3. Human review

Humans remain essential for ambiguous cases, taste, subjective quality, novel failure modes, high-risk decisions, expert-domain evaluation, and calibrating automated graders. But humans aren't perfectly consistent either. Two senior lawyers may disagree on whether a clause is acceptable; two designers on whether a generated interface is good. That disagreement isn't necessarily noise. Sometimes it reveals that your rubric is underspecified. For important domains, use multiple reviewers, measure disagreement, reconcile difficult cases, and refine definitions. Once an automated grader is reliable enough for a task, human review can become sampled rather than exhaustive, with humans focused on validating the automated system, discovering new failure modes, and spotting rubric drift.

Non-Determinism Means One Run Isn't Enough

Traditional tests generally behave the same way every time. Generative models may not. Suppose Model A passes 82 of 100 cases and Model B passes 85. Is B definitely better? Run them again and you might get A: 84, B: 82. The difference may be noise. For important evaluations, especially agentic workflows, repeated runs tell you about reliability. Two useful concepts:

For some applications, consistency matters more than peak capability. A PM should distinguish between “can the system do this?” and “can the system be trusted to do this repeatedly?” Those are different product properties.

Evaluating Agents: Don't Only Grade the Final Answer

Agents make evaluation harder because the output isn't only text. An agent may search, retrieve, reason, call tools, modify systems, retry, ask for clarification, or trigger downstream processes, and there may be many valid ways to solve the same task. To refund a customer, one valid trajectory is retrieve customer → retrieve invoice → check policy → confirm → process. Another is retrieve invoice → identify customer → check policy → confirm → process. Both may be correct. If your evaluator requires exactly one predetermined sequence, it may penalize valid behavior. For agents, evaluate several layers:

You don't need to punish every alternate path, but you do need to inspect the path when hidden behavior can create risk. Evaluate outcomes rigidly and trajectories thoughtfully.

Illustrative Scenario: The Support Agent

Using Evals for Model Selection

One of the most valuable uses of evals is deciding which model to use. The wrong question is “which model is the smartest?” A better one: “which system configuration produces the best outcome for our workload under our quality, reliability, latency, and cost constraints?” Run the same set against each candidate and score dimensions separately.

Now the trade-offs are visible. Model B has the highest task success but is expensive and slow. Model C is cheapest and most faithful but weaker on task completion. Model A may provide the best balance. The answer depends on the product. There is no universal winning model.

Evaluate the System, Not Just the Model

When teams say they're “evaluating a model,” they're often evaluating a much larger system. The output may depend on the model, model version, prompt, system instructions, retrieved context, chunking, reranking, tools, tool descriptions, memory, orchestration, retry policy, reasoning configuration, and the interface. A smaller model with excellent retrieval can outperform a much larger model with poor context. A model with slightly weaker reasoning may outperform another because it follows your tool schema more reliably. A different prompt can matter more than switching models. The unit of evaluation should usually be the product configuration, not merely the foundation model. That's what you're actually shipping.

Set Release Gates, Not Just Scores

A dashboard full of eval numbers is useful. A release policy is more useful. Define what must be true before a configuration can be promoted. For example, ship only if task success improves or stays statistically equivalent, faithfulness doesn't regress by more than 2 points, no critical safety failures appear, p95 latency stays below 5 seconds, cost per successful task stays below the agreed limit, and performance on high-value customers doesn't regress. The exact thresholds depend on the product. The important part is that the rules are agreed before the team sees the result. Otherwise every model decision becomes “yes, it's worse on accuracy, but look how much better the tone is,” or “yes, cost doubled, but this is the newest model.” Release gates reduce post-hoc rationalization and make trade-offs explicit.

Segment the Results

Aggregate scores hide serious problems. Suppose task success improves from 84% to 87%. Looks good. Now break it down:

The headline improvement is real. So is the enterprise regression. If enterprise accounts represent the majority of your revenue, the product decision may be obvious. Useful segmentation can include customer tier, workflow, language, geography, task complexity, document type, industry, tool, risk class, and new versus experienced users. The average is useful; the distribution is where many of the important product decisions live. Always ask who improved and who regressed.

Version Everything & Don't Overfit

An eval result is only useful if you can reproduce it. Version the model, model parameters, prompts, tools, retrieval configuration, chunking, reranking, datasets, grader prompts, judge models, application code, and knowledge-base version. Otherwise teams end up saying “the newer prompt from sometime in March seemed better with the previous model,” which isn't a result. A better record: “Configuration 2.14 improved task success from 81.6% to 86.3%, with no significant regression in faithfulness, cost, or latency.” Now it can be reproduced, and if production quality drops later, the team knows exactly what changed.

And once a score becomes important, teams naturally optimize against it, until it becomes too useful. If engineers repeatedly see every test case, they may build behavior that specifically handles the benchmark. The score rises; real-world performance doesn't. Counter this with held-out cases, periodically refreshed capability sets, new production traces, blind human review, multiple suites, adversarial cases, and segmentation. The purpose of the eval is not to get a high eval score. It's to estimate product quality.

The Eval Flywheel

A practical evaluation system is a continuous loop: define the job (prefer “resolve the billing issue correctly without violating policy” over “generate a good answer”); identify failure modes from traces, support data, edits, experts, analytics, complaints, and abandoned flows; build the eval set from normal, known-failure, difficult, high-value, and high-risk cases; define independent dimensions rather than one number; choose the right graders; run on every material change; connect offline scores to product metrics; and feed production failures back in. The suite is never finished.

Evals as Product Specifications

Traditional PRDs often say “the assistant should provide helpful, accurate responses.” That sounds reasonable and is almost impossible to test. An eval forces the requirement to become concrete. For billing support: ≥95% policy correctness, ≥99% correct escalation, zero unsupported refund-policy claims in the critical regression suite, sub-5-second p95 latency, under 10% user edit rate after launch. Now “good” has meaning. People may argue about the threshold, which is healthy, but the requirement has become observable.

This is why evals increasingly become a shared specification between product, engineering, ML, data science, design, operations, domain experts, and leadership. The discussion changes from “I think this version is better” to “it improves task success by five points but violates the faithfulness gate.” That is a much more mature product conversation.

What the PM Should Own

Engineering understands the system, ML understands model behavior, data science understands measurement, domain experts understand what correct means inside the business, design understands the experience, and product connects those pieces. The PM should be able to answer: what job are we evaluating, which workflows and users matter most, what counts as success, which failures are unacceptable, which metrics should improve, which must never regress, what trade-offs are acceptable, how reliable the system needs to be, how offline quality connects to real product behavior, and which model or configuration to ship. Those are product decisions. That's why eval literacy is becoming a core PM skill.

Checklist You Can Use Monday

• A clear definition of the job this AI feature completes?

• An explicit list of top failure modes?

• Eval set includes real production traces or high-fidelity simulations?

• Common, difficult, high-value, and high-risk cases represented?

• Capability separated from regression evals?

• Some cases held out from everyday tuning?

• Important dimensions scored independently?

• Deterministic behaviors checked with code, not an LLM judge?

• LLM judges calibrated against human labels, with disagreement mapped?

• Guardrails treated as explicit gates, not buried in an average?

• Agent evals check actual state changes and tool outcomes?

• Important cases repeated when reliability matters?

• Results segmented by important customers, workflows, or risk classes?

• Suite run before every material change, with config and dataset versioned?

• Production failures continuously feeding new cases, connected to a real product metric?

Common Mistakes

Key Takeaways

Ask Your DS / Eng Team