Evals before launch: how we take an LLM feature to production

Most AI features look finished long before they are. Someone wires a model up to a few documents, asks it five questions, gets five good answers, and the demo goes well. Then real users arrive with questions nobody wrote down, the source documents change, a model version is retired, and the answers quietly get worse. Nobody notices until a customer does.

The gap between that demo and a feature you can rely on is mostly one thing: evaluation. Before we call an AI capability ready for production, we want a repeatable way to answer "is it good enough?" and to keep answering it after launch. This is how we approach it.

Start with what "good" means for this workflow

An eval is only as useful as the definition of success behind it. So before writing prompts, we agree with the people who own the workflow what a correct result looks like, and what a harmful one looks like.

For a document extraction feature, that might be: every required field is present, dates and amounts match the source exactly, and anything the model is unsure about is flagged rather than guessed. For a question-answering assistant over internal knowledge, it might be: the answer is supported by the cited source, it says "I don't know" when the source doesn't cover the question, and it never invents a policy.

We write these down as acceptance criteria, in the same spec that covers the rest of the feature. If we can't agree what good looks like, the feature isn't ready to build.

Build the eval set from real examples

The eval set is a collection of inputs paired with what the right output should be. We build it from real material wherever the client can share it: actual documents, real support tickets, questions people have asked. Invented examples tend to be cleaner and easier than reality, which makes the results look better than they are.

A useful eval set covers:

• The common cases, weighted roughly the way they appear in real use.
• The awkward cases people already know about: scanned PDFs, tables, forms filled in by hand, questions that span two documents.
• The cases where the right answer is "no" or "I don't know", because a system that always answers will eventually answer wrongly with confidence.
• Anything with a cost if it goes wrong, such as amounts, dates, names or compliance statements. These get checked most strictly.

It starts small, often a few dozen examples, and grows. Every time a problem shows up in testing or production, the example that caused it goes into the set, so the same mistake is caught automatically next time.

Score it the same way every time

For each example we record whether the output passed, and why not if it didn't. Some checks are simple code: does the extracted amount match exactly, is the JSON valid, did the answer cite a source? Others need judgement, like whether an answer is actually supported by its source. For those we use a mix of a second model acting as a grader, with its own written rubric, and people spot-checking the grader.

The output is a scorecard: pass rates by category, the failures listed, and cost and response time per request. That scorecard is what we review with the client. It turns "it seems pretty good" into a decision you can make.

Treat the eval as a regression gate

Once the scorecard is at the agreed level, it becomes a gate. Any change to a prompt, the retrieval setup, the model or its version runs the eval again before it ships. If quality drops in a category that matters, the change doesn't go out.

This matters more than it sounds. Model providers update and retire versions on their own schedule. A prompt tweak that fixes one case often breaks two others. Without a gate, every change is a guess. With one, you can switch models to save cost, or adopt a better one, and know exactly what changed.

It is the same principle we apply to the rest of the codebase, where every change runs tests, type checks and security scanning, and an engineer reviews it before merging. AI features get the same treatment, with evals standing in for most of the tests.

Put people where mistakes are costly

Evals tell you how often the system is right. They don't make it right every time. So for each feature we decide, with the client, where a person stays in the loop.

That's usually driven by the cost of a mistake. A suggested reply that an agent reads before sending can tolerate occasional misses. An extracted invoice amount going straight into a finance system cannot. There, we route low-confidence results to a review queue, show the source next to the extracted value, and make approving or correcting it quick. Those corrections also feed back into the eval set.

Watch it in production

Launch is where the eval set stops being the whole picture. Real traffic brings inputs nobody thought of, so we monitor:

• Quality signals: how often people correct, reject or re-ask, and samples of live outputs scored against the same rubric.
• Cost: model spend per request and per month, against the budget agreed at the start.
• Latency: how long users wait, especially at busy times.
• Drift: changes in the kind of input coming in, such as a new document template, which often shows up before quality drops.

We review these on a set schedule, not just when something breaks, and new failure cases go back into the eval set.

What this looks like in a project

In practice, evals aren't a separate phase at the end. The first version of the eval set is written during the prototype, so we can say early whether an approach is working. It becomes the regression gate during integration, and a monitoring baseline after launch. By the time a feature ships, the client has a scorecard, a way to re-run it, and a clear view of where people review results and why.

It's also the honest way to decide that AI isn't the right tool. Sometimes the scorecard stays below the bar however much we improve the approach. That is useful to learn in week two rather than month six.

---

If you have an AI feature that works in a demo but you're not sure it's ready for real users, that's the conversation to start with. We can look at what you have and what it would take to measure it properly.