A model can improve on the dashboard while becoming worse for the people who most need it. That does not make averages dishonest. It means an average answers a narrower question than a product team sometimes assumes.
Consider an illustrative evaluation with 900 routine requests and 100 complex requests. A system that handles 98% of the routine cases and 50% of the complex cases achieves 93.2% overall. The arithmetic is correct: 932 successful requests out of 1,000. The product interpretation depends on what happens to the 68 failed requests.
Start with the decisions behind the errors
A mistaken suggested label, a misleading answer, and an unauthorized action have different consequences. Counting them identically can obscure that distinction. Before optimizing a metric, describe the failure categories and decide which require an absolute release gate rather than a trade-off against successes elsewhere.
Do not automatically equate business importance with customer revenue. Accessibility, contractual commitments, safety, and underserved user groups can matter even when their traffic is small. A prioritization decision should be explicit enough to challenge.
Use slices without manufacturing certainty
Break down outcomes by dimensions connected to the task: language, input length, document type, workflow stage, or source quality. Keep the sample count next to every rate. A weak score on a tiny slice is a reason to investigate, not proof of a stable performance gap.
The worst observed slice is useful as an alert, but it is a noisy statistic. If a team inspects enough small groups, some will look unusually bad by chance. Predefine important slices, collect more examples where needed, and distinguish exploratory findings from release criteria.
Cost weighting is a model of consequences
A cost-weighted error metric can express the difference between a harmless retry and a costly manual correction. Its weights are assumptions, not facts produced by the classifier. Write down who chose them, what evidence supports them, and how the decision changes under plausible alternative weights.
Keep raw counts and examples available alongside the weighted score. Otherwise, a team may improve the score by shifting errors into a category with a convenient weight. The metric is supposed to clarify the consequences of errors, not make them disappear.
Accuracy and calibration answer different questions
Accuracy describes how often predictions are correct under a particular decision rule. Calibration asks whether predicted probabilities correspond to observed frequencies. If cases assigned similar probabilities succeed much less often than expected, those probabilities should not drive an automatic approval threshold without further work.
Evaluate calibration on data separate from model fitting and check whether it holds across important populations. A language model writing “90% confident” is not, by itself, supplying a validated probability. Interfaces should not turn that phrase into a precision-looking badge.
Build a decision-sized report
- Show the overall outcome and its denominator.
- Show predefined slices, sample sizes, and uncertainty where it can be estimated.
- List critical failures separately from compensable errors.
- Include the operational cost of review, retries, and abandoned tasks.
- Compare against the current system, including a non-ML baseline where relevant.
The aim is not to replace one headline number with twenty more. It is to make the next decision easier: which failure should be fixed, which users need protection, and whether the improvement is real enough to ship. This connects directly to choosing product metrics and designing a useful abstention path.
Count failed tasks, not automatically failed users
In the 1,000-request example, 18 routine requests and 50 complex requests fail, totaling 68 failed requests. That does not establish 68 distinct affected users: one person may submit several requests, and the consequence of failure can differ. Report the request-level arithmetic, then measure affected users or accounts separately if the product question requires it. Preserve the distinction when aggregating repeated trials of the same task.