Designing a Latency Budget for AI Features

An AI feature can feel slow even when the model call is reasonably fast. Authentication, retrieval, reranking, tool calls, queueing, and post-processing all contribute to the wait. Optimizing the model alone can leave the actual bottleneck untouched.

I would start with the user's workflow: when does the wait begin, when can the user do something useful, and when is the task actually complete? Those are not always the same timestamps.

Define the experience before the budget

An inline suggestion, an interactive research request, and an overnight report have different acceptable waits. Do not reuse one latency target simply because all three call a language model. Describe what users can do while waiting and whether cancellation or partial results are useful.

Streaming can reduce the time before text appears, but the first token is not necessarily the first useful result. A user may still need the complete answer and its sources before taking action. Measure both the intermediate experience and completion.

Trace the whole request

Record the stages of a representative request under a shared trace identifier. Identify serial dependencies, parallel work, retries, and queueing. Segment by task type and input size so a large-document workflow does not hide inside the same average as a short lookup.

Look at the distribution rather than only the mean. Slow-tail requests are part of the product experience. Do not add stage-level 95th percentiles and label the result the end-to-end 95th percentile; the end-to-end distribution needs to be measured directly.

Make a provisional budget explicit

For an illustrative synchronous workflow with a two-second target, a team might reserve 200 milliseconds for request handling, 300 for retrieval, 1,200 for generation, and 300 for validation and contingency. These are hypothetical planning allocations, not measured benchmarks or universal recommendations.

The point of the budget is to reveal trade-offs. If one stage routinely consumes the whole allowance, the team must change the stage, the workflow, or the target. It cannot solve that conflict by relabeling waiting time as “thinking.”

Remove unnecessary work before removing safeguards

Check for repeated retrieval, duplicate model calls, excessive context, and sequential work that has no real dependency. Cache stable intermediate results when correctness and access boundaries allow it. Include relevant tenant, permission, source-version, and configuration context in cache design.

Model routing or a smaller model may help for bounded subtasks, but it needs an outcome comparison. Reducing verification to meet a latency target can create a faster incorrect answer. Keep critical authorization and action checks outside that trade-off.

Bound retries and wasted work

A timeout followed by several retries can exceed the original user expectation while increasing load. Give the request an overall deadline and propagate cancellation where dependencies support it. Define which operations are safe to retry, especially when a tool may have performed a write before timing out.

For long tasks, an asynchronous workflow with explicit status and a retrievable result may serve users better than a long-held request. The interface should explain what happens if the user leaves and how a failed job can be resumed or restarted.

Evaluate speed and usefulness together

Compare completion, quality, abandonment, and operational cost before and after the change. A faster answer that creates more correction work is not necessarily a faster task. Keep the critical-path trace next to the product outcome, and the next optimization becomes easier to choose.