Product Metrics That Change a Decision
A metric becomes useful when a team knows what it would do differently if the number changed. Without that connection, a dashboard can become a monthly ritual: the line moved, somebody supplied a story, and the roadmap stayed the same.
I would begin with the product's intended outcome and work backward to the evidence needed for a decision. The result is usually a smaller, more interpretable set of measures than a collection of every event the application can emit.
Separate activity from completion
Imagine a hypothetical self-service export feature. Export-button clicks describe activity. A usable file delivered to the right user describes a service outcome. A successful downstream analysis may be the broader user outcome, though it is harder for the product to observe directly.
Name those boundaries rather than pretending the easiest event is the final goal. A rise in clicks can mean growing use, repeated failures, or a confusing interface. The same line can support incompatible stories unless the surrounding behavior is measured.
Build a small metric hierarchy
Use an outcome measure to describe whether the task succeeds. Use diagnostic measures to locate friction, such as invalid requests, queue time, or failed downloads. Add guardrails for consequences that should not deteriorate, such as incorrect data exposure or support burden.
For the export example, the core measure might be the fraction of eligible requests that produce a retrievable, valid file. A diagnostic might be time spent waiting for generation. A guardrail might track requests that include data outside the user's authorized scope, with any occurrence requiring immediate investigation rather than averaging it away.
Write the denominator in plain language
Does completion mean completed requests divided by all initiated requests, all valid requests, or all sessions that viewed the page? Each definition answers a different question. State the eligibility rule and keep excluded cases visible.
Define the observation window too. A request that completes tomorrow may look like a failure in today's snapshot. Use mature cohorts when comparing outcomes, and distinguish pending work from completed failures.
Check instrumentation before interpreting behavior
Verify event names, duplicate handling, identifiers, and the sequence that connects initiation to completion. An event emitted on button click is not evidence that the server accepted the request. An event emitted before a database transaction commits may overcount success.
Test representative flows, including retries and failures. When instrumentation changes, annotate the break in the series. A measurement change should not be celebrated as a product improvement.
Choose a review question, not only a target
A review might ask whether failed exports are concentrated in one file type, whether users return because the first file was unusable, or whether a faster generation path increases support work. That points the team toward a specific investigation.
Targets can focus effort, but a target without a failure analysis can encourage local optimization. Faster average completion may be achieved by rejecting difficult requests. Pair the outcome with coverage and the experience of excluded users.
Do not infer causality from every movement
A before-and-after change may reflect seasonality, a different user mix, or an unrelated release. Use an appropriate experiment when causal attribution matters and the conditions support one. Otherwise label the observation honestly and combine it with other evidence.
A useful metrics review ends with a decision, an owner, and a reason. It may conclude that more evidence is needed. See working with limited experimental traffic and reading aggregate model metrics carefully.