Evals, part 2: choosing the right ones for your product

Part two of three, on measuring the quality of AI products.

Start with the product, not the metric

Part one argued that GenAI products fail silently, and that an eval is the only way out of the whack-a-mole loop. This is how you work out which evals your product actually needs.

The most common way I see eval work go sideways is that a team reaches for a metric they read about (hallucination rate, some benchmark) before they’ve figured out what kind of product they’re even building. The metric drives the work instead of the product driving the metric.

Flip it. Here’s the order I use.

The evals stack

Build it bottom-up, where each layer constrains the one above it:

1. Type of AI product. The foundation. This decides everything else.
2. Evaluation timing. Offline, online, or real-time.
3. Types of evaluators. Human annotators, LLM-as-a-judge, code, heuristics and rules.
4. Types of metrics. Accuracy, precision and recall, F1, pass rate, latency, cost, and so on.

Start at the bottom. A conversational assistant, a RAG Q&A tool, an autonomous agent, and a copilot need different evals, so if you pick metrics before you’ve named the product type, you’re guessing.

The 7 archetypes

When I was putting this together, I found that every AI product in the portfolio fell into one of seven buckets, each defined by a single core eval question.

#	Archetype	Core eval question	Examples
1	Grounded generation	Is it faithful to the source and useful?	Property Q&A, comparison tools, long-form research
2	Verifiable / reference-based	Is it correct vs ground truth?	Ranking, extraction, forecasting, code and SQL
3	Open-ended generation	Is it high-quality, on-brand, safe?	Marketing copy, property descriptions, generated media
4	Agentic / task completion	Task done correctly and safely?	Booking and trip-planner agents, coding agents, voice agents
5	Perception / signal	Did it perceive accurately?	Photo tagging, image scoring, OCR, call transcription
6	Integrity, safety and detection	Catch the bad, don’t over-block?	Moderation, PII and injection defense, fraud
7	Decision / optimization	Did the action maximize reward?	Dynamic pricing, offer optimization, bandits and RL

Two things worth calling out. First, archetype 7 was where we were already strong: dynamic pricing, multi-arm bandits, in-house optimization systems. Decisions we knew how to evaluate. The muscle still being built was on 1 through 6.

Second, real products span buckets. A booking agent (4) wraps grounded generation (1) and retrieval (2); you evaluate the trajectory on top of each part’s own eval, not instead of it. And archetype 6, integrity, shows up inside nearly every other one.

Once you know your archetype, the evaluators and metrics mostly fall out of it. That’s the point of the map.

The one distinction that shapes everything

Before you can pick evaluators, answer one question: is there a right answer to compare against? Most products use both, but the split matters.

Reference-based. A vetted “golden” answer exists for each input, and you score by comparing. “What time is check-in at the Hilton Miami?” is 3 PM. You can put that in a golden dataset, call the agent, and check the answer matches. Objective, cheap, repeatable, but it needs labeled data and can’t judge open-ended quality.

Reference-free. There is no golden answer, so you judge the output on its own merits against a rubric. Best for open-ended, generative, subjective work. More powerful, but biased, costlier, less reproducible, and it needs human calibration.

The tell: if you can imagine building a golden dataset, you’re reference-based. If you can’t picture what the single correct output would even be, you’re reference-free.

Three products, three shapes

A comparison assistant (grounded generation, mostly reference-based). You’re on the search results page, pick up to five properties, and go back and forth with it, so it’s multi-turn. The facts are checkable: if a property is tagged family-friendly in internal data, you can build a dataset with four properties that aren’t and one that is, ask which is family-friendly, and score the answer. Easy to build a repeatable offline eval.

A long-form research tool (open-ended generation, reference-free). You give it a long brief (“scenic mountain cabin for five, fireplace, away-from-it-all”) and it generates a magazine-style experience in a single shot. There’s no perfect answer to compare against, so you evaluate the shape. Is it comprehensive? Is it well-structured? When it makes a claim, is it cited and grounded? You can keep a golden set of queries you want to support, but you won’t have golden outputs. Hard to eval offline.

An internal analytics assistant (verifiable / reference-based, and a case for real-time). This one searches company data and answers business questions, where accuracy matters more than latency. Someone asked it to compare conversion for vacation rentals across two of our booking surfaces. That’s a trick question. On one surface, vacation-rental traffic is classified as lodging in the clickstream, so a naive query quietly compares that surface’s entire lodging category (hotels plus rentals) against the other surface’s rentals alone, which is not apples to apples. It answered confidently, with right-looking numbers, and it was wrong on relevance. It answered a question the user didn’t ask. That’s exactly the kind of miss a real-time check could catch before the answer ever reaches the user.

The takeaway: name your archetype first, decide reference-based vs reference-free second, and the rest of the stack gets a lot easier.

Offline, online, real-time: when your eval runs

Once you know your product archetype, the next layer of the stack is timing: when the eval runs relative to the user. There are three answers, and mature products use all three.

The real question that separates them is simple: does the eval run inside the request path and gate the answer before the user sees it?

Offline, before you ship

Offline evals run pre-deploy, in CI, on fixed datasets. They never touch production, so timing and latency don’t matter at all.

This is controlled testing against curated golden datasets, usually built from the PRD, plus simulated scenarios. It’s typically reference-based: every test case has a vetted, human-approved answer to score against. Offline evals tell you how the agent performs before you turn it on, and they’re your regression net when you change a prompt or a model.

The loop is:

create golden dataset → invoke the agent → score the outputs.

One nuance: “offline” doesn’t have to mean real users. You can run a golden dataset that actually calls your live agent on a schedule (daily, weekly, on every PR) and gate the CI/CD pipeline on it.

Online, after you ship

Online evals run in production, on sampled live traffic, and they’re scored asynchronously after the fact, so they don’t add any latency to what the user experiences.

Their job is to catch what your test cases never anticipated. The PRD is a guess about how people will use the product; real traffic is the truth, and it’s full of cases nobody wrote down. Online evals surface drift, measure real-world quality, and feed A/B tests. Because there’s no vetted answer for a live trace, they lean reference-free: the evaluator judges the output on its own merits.

The loop is:

publish the trace → score and calibrate with human annotators → dissatisfaction seeds new evaluators.

That last step is the important one. Production failures become your next offline dataset. Offline validates before you ship; online watches after you ship; and what online finds flows back into offline. It’s one system, not two.

Real-time, inside the request path

Real-time (inline, in-the-loop) evals run synchronously, inside the request path, and they gate or correct the answer before it’s displayed. This is the only mode that can guarantee a user never sees an unvalidated answer.

The trade-off is explicit: you accept added latency to avoid a wrong answer. Accuracy beats speed. Typical mechanisms are a verifier or critic pass, a groundedness check, SQL reconciliation, or a guardrail classifier that can trigger a retry or flag the response.

We hadn’t implemented this, but the analytics assistant is the product that made the case for it. Remember the conversion-rate trick question: it answered confidently and got the relevance wrong. For a tool where accuracy matters more than latency, a real-time check could verify the answer against the user’s actual intent before responding, catch the mismatch in the moment, and either fix it or honestly say “I’m not sure” instead of shipping a confident wrong answer.

Don’t pick one, layer them

Eval type	When it runs	Blocks response?	Best for
Offline	Pre-deploy / CI, fixed datasets	No	Regression, benchmarking, validating a change
Online	Production, sampled live traffic	No (scored async)	Drift, unanticipated cases, A/B
Real-time	Synchronously, in the request path	Yes	Guaranteeing the user never sees a bad answer

These aren’t mutually exclusive. A mature product runs offline to ship safely, online to learn from reality, and real-time on the paths where a wrong answer is genuinely costly. Start with offline, add online as you get real traffic, and reach for real-time on the high-stakes flows.

That is the map. Part three is what a year of actually building these taught me, starting with four principles I would give you before you write a single evaluator.