How to evaluate an LLM feature before you ship it

Written By: BrainSoft In AI & Research

Teams ship LLM features on vibes far too often. Someone tries twenty prompts in a chat window, the output looks good, and the feature goes to production where it starts failing on inputs nobody imagined. The demo was never the hard part. Knowing whether the thing actually works, and for whom, is.

Before you ship, build a small evaluation set from real inputs, define what a correct answer looks like, and score every prompt or model change against it. Run it in CI, watch cost and latency alongside quality, and only then decide whether the feature is ready.

Start with a real eval set, not sample prompts

The single most useful thing you can do is collect inputs from the people who will use the feature. Support tickets, search logs, form submissions, whatever already exists. Aim for 50 to 200 cases. Fewer than that and a single failure swings your score by two percent. More than a few hundred and you will not maintain it.

Store each case with the input, the expected output or the properties a good output must have, and a short note about why it matters. Keep it as JSON in the repo so it lives next to the code and gets reviewed in pull requests.

[
  {
    "id": "invoice-date-014",
    "input": "Invoice dated 3/4/2024, due in 30 days",
    "must_contain": ["2024-04-03"],
    "must_not_contain": ["2024-03-04"],
    "note": "EU date format, team keeps getting this wrong"
  }
]

Deliberately include the ugly cases. Empty strings, very long inputs, mixed languages, prompt injection attempts, and inputs that should be refused. If your feature cannot say no, that is a finding, not a surprise for your users.

Pick metrics you can actually compute

Not every output needs a human judge. Split your checks into three buckets and use the cheapest one that works.

• Deterministic checks. Does the JSON parse? Does it match the schema? Does it contain the required date? These run in milliseconds and catch most regressions.
• Reference comparison. When you have a known good answer, compare with exact match, token overlap, or embedding similarity. Rough, but cheap and stable.
• Model-as-judge. For open-ended outputs, ask a stronger model to grade against a rubric. Useful, but validate the judge against your own labels first, or you are measuring the judge.

Whatever you choose, log the raw output for every case. When a score drops you want to read the actual text, not just the number.

Treat prompts and models as versioned code

A prompt change is a code change. So is swapping the model behind it. Both need to go through the same eval run before merge. Wire it into CI so a pull request that touches the prompt, the model name, or the parsing layer triggers the suite and posts the diff.

python -m evals.run \
  --set evals/invoices.jsonl \
  --prompt prompts/extract_v3.txt \
  --model gpt-4o-mini \
  --out results/v3.json

Keep a baseline result file in the repo. Compare against it and fail the build if pass rate drops more than a point or two. Two points is noise on a small set; ten is a real regression.

Do not forget the boring parts: cost per request, p95 latency, and token counts. A prompt that improves accuracy by three percent but doubles latency is a trade-off someone has to make on purpose, not by accident.

Plan for production drift

Your eval set goes stale. Users type things you never imagined, the provider updates the model under the same name, and the world moves on. Set up a feedback loop before launch: thumbs up and down, a way to flag bad outputs, and a weekly pass where someone promotes the worst real cases into the eval set.

If the feature is customer-facing and failure is expensive, add a human review step or a confidence threshold that routes low-confidence outputs to a person. That is not a weakness in the design. It is how you find out what the model does not know.

For teams that want help building this kind of harness, or wiring it into an existing pipeline, get in touch. We do this a lot.

Frequently asked questions

#AI & Research