A fluent answer is easy to demonstrate. A dependable application is harder to demonstrate because the unit of success is not a sentence: it is a user completing a task under real constraints. An answer can be well written, grounded in a document, and still be useless if it arrives too late or answers the wrong question.
I would start an evaluation plan with a short product contract: who the application serves, what it is allowed to do, what evidence it needs, and what happens when it cannot finish. The evaluation should test that contract, not reward the model for sounding confident.
Define success before collecting scores
Consider a hypothetical assistant that answers employee policy questions. Success includes selecting the policy that applies to the employee, using the effective version, citing the relevant passage, and declining requests outside its permissions. Text similarity to a reference answer captures only a small part of that job.
Separate checks into three groups: task outcome, system behavior, and operational cost. Outcome checks ask whether the response is correct and useful. Behavior checks cover authorization, tool use, and abstention. Operational checks cover latency, retries, and cost per completed task. Do not combine these into one number before looking at the individual failures.
Keep development and holdout cases separate
Use development cases while changing prompts and retrieval. Reserve a separate holdout for release decisions, and track how often it has been inspected. Repeatedly tuning against a supposedly frozen test set makes it another development set.
Include representative requests, known incidents, ambiguous inputs, and cases the product should not answer. Where production examples are used, obtain the appropriate permission and remove unnecessary personal or confidential content. Synthetic cases are useful for coverage, but their realism needs checking against actual tasks.
Use the cheapest trustworthy check
Validate schemas, required fields, citation destinations, and forbidden actions in code. Use a model judge only for criteria that need interpretation, such as whether a summary preserves an important qualification. Give the judge a specific rubric and test it against independently reviewed examples.
There is no universal number of human spot checks or judge-agreement percentage that makes an evaluation reliable. Review enough examples to expose consequential disagreements, and examine those disagreements by failure type. A judge that consistently misses one serious error can look reassuring in aggregate.
Make uncertainty visible in the report
Report denominators alongside rates. A perfect score on three examples is not equivalent to a perfect score on three hundred. Break down results by meaningful task, language, and document categories, while flagging slices too small to support a strong conclusion.
For failures with different consequences, define weights with product and domain owners rather than choosing an impressive-looking multiplier. Keep the unweighted counts visible. A weighted average should not conceal an unauthorized action or another release-blocking event.
Turn the evaluation into a release decision
- Compare the candidate with the current release on the same versioned cases.
- Show changed examples, not just the difference between two averages.
- Run repeat trials where nondeterminism could change the conclusion.
- Define acceptable regressions and non-negotiable failures before seeing the result.
- Connect offline checks to a limited rollout, monitoring, and a rollback owner.
The useful deliverable is a defensible decision: ship, investigate, or stop. For the next step, see how to release prompt changes and why averages need context.
Specify the uncertainty method
When rates come from independent binary tasks, a binomial interval is a possible summary; repeated outputs from the same task are not independent tasks. For a paired candidate-versus-baseline comparison, preserve pairing and consider a task-level resampling or a paired analysis appropriate to the outcome. State the unit, sample size, repeats, and method in the release report. Keep judge agreement separate from application success, and review critical disagreement examples rather than using agreement as a universal certification threshold.