Evals, part 1: when “it looks fine” stops working

Part one of three, on measuring the quality of AI products.

For years, shipping an ML product had a comfortable rhythm to it. You trained on 80% of your data, held back 20%, and tested against it. Same input, same expected output. If the model said “watch Movie B” and the label said “Movie B,” you were right. Quality was a number you could compute.

Then we started building GenAI products, and that comfortable rhythm broke.

The thing that changed

Ask a GenAI product to summarize a movie and you’ll get one answer. Ask it again and you’ll get a different one. Both can be correct. The output is probabilistic: the same prompt can produce many acceptable responses, and there is no single “right” cell in a spreadsheet to compare against.

So the old question, did it match the label?, stops working. The new question is harder: how do you measure quality when there are many good answers and no answer key?

That’s not an edge case anymore. This is drawn from a few years inside a large travel marketplace, and as AI got embedded across more of what we built, most products ended up with some component where “correct” was a distribution.

What does “good” even look like?

Three real interactions with a property Q&A assistant make it concrete.

• “Does the hotel allow late checkout?” It said yes, for a fee, CAD 50. I can verify that against the tool call it made. It’s faithful. I can score it. Easy pass.
• “Do they have EV chargers?” It understood EV means electric vehicle, noted parking is CAD 25, cited the detail from a guest review, and confirmed the chargers. Also good, also checkable.
• “Find some family-friendly restaurants nearby.” It returned restaurants. A naive eval passes this. They are restaurants, they might be family-friendly. But to actually pick a restaurant you need to know where it is, how far, what the menu looks like, whether there are photos. None of that was there. The answer looks fine on the surface, but it is not helpful and they quietly leave without booking.

That third one is the whole problem in one interaction. AI products fail silently. The response is fluent, confident, and shaped like a good answer, so nothing trips an alarm, and a traveler bounces without ever telling you why.

What travelers are actually asking

Underneath every interaction, a traveler is really asking four things:

1. Do I like this answer? (The restaurant reply was technically fine and still unsatisfying.)
2. Is the AI answering my question? (Is it relevant?)
3. Can I trust this answer?
4. Is the answer complete and precise?

That last one is newer and sneakier than it sounds. People stack multiple questions into one message, and the model happily answers half of them and drops the rest. We see this a lot. If you’re not watching for it, you’ll never know it’s happening.

Life without evals is whack-a-mole

Without a systematic way to measure this, here’s the loop you’re stuck in: ship a prompt, something feels off, panic, hack the prompt, ship again, and you have no idea whether you just fixed the problem, moved it, or made it worse. You’re swinging at moles as they pop up, one at a time, forever.

An eval is the way out. The definition I use: a systematic measurement of the quality and effectiveness of an AI system. “Systematic” is doing a lot of work in that sentence. It means measuring from every side of the product, not spot-checking whatever you happened to notice.

Two questions tell you whether you have evals or not:

• How do you know your agent works?
• How well does it work?

If you can’t answer both with a number you trust, you’re playing whack-a-mole.

You can’t do this with humans alone

The gold standard is still a human, ideally a domain expert or the PM, reading the output and checking it against the PRD. Early in a product’s life, that’s exactly what you should do. But it’s slow, expensive, and doesn’t scale past a few hundred examples.

So you scale human judgment with AI: LLM-as-a-judge and programmatic checks that are fast, cheap, and repeatable. The catch is that a judge is a model too, so you have to keep calibrating it against humans. Does the judge’s “pass” match a human’s “pass” on the same response? That gives you a confidence score in your automated judgment.

Think of it like a garden. You don’t calibrate once and walk away; you keep tending it, periodically checking, trimming when it drifts. The principle to hold onto:

Keep humans in the loop. Use tools and AI to scale human review, not replace it.

That’s the “why.” The “how” starts with a framework for figuring out which evals your specific product even needs, which is where part two picks up.