Before You Add an LLM, Build the Simplest Useful Baseline
A product brief says “add AI search.” Before selecting a model, I would ask what the current search fails to help someone do. If the answer is finding a record by its exact identifier, the first experiment probably does not need a generative model. If the answer is comparing contradictory reports, the problem is different.
This is not an argument against LLMs. It is an argument for giving them a fair comparison. A feature should earn its operational complexity by making an important task meaningfully better.
Write the job without the technology
Replace “build an intelligent assistant” with a task statement: a support analyst needs to find the current resolution procedure for a specific incident without opening several outdated documents. That statement identifies a user, an outcome, and a source of friction.
It also leaves room for several solutions: better document ownership, metadata filters, search, a guided decision tree, or a conversational interface. Naming the technology too early can hide the simpler intervention.
Build a baseline the user could keep
A useful baseline is not deliberately weak. For the support example, it might combine keyword search, effective-date filtering, clear document titles, and a link to the relevant section. It could be less impressive in a demonstration and more useful during an incident.
Where a task is poorly understood, run a small manual service first. Clearly disclose the human involvement. Record what information was required, where judgment entered the process, and which exceptions made the work slow. That is discovery evidence, not proof that automation is ready.
Compare complete workflows
Give both approaches the same representative tasks. Measure whether users reach the correct outcome, how long it takes, what they need to verify, and how they recover from a failure. A faster first answer can still produce a slower completed task if the answer needs extensive checking.
Include requests that should produce no answer, records with similar names, and cases outside the supported scope. Keep the comparisons fair: the baseline and the model should have equivalent access to relevant information and neither should receive hidden hints.
Look for the part where a model helps
An LLM may add value by translating an informal request into a structured query, summarizing retrieved passages, or drafting a response for review. Those are different jobs. A result that supports one does not automatically justify an autonomous end-to-end agent.
For the support workflow, query reformulation might help while free-form answer generation introduces unnecessary verification. That suggests a smaller intervention: improve retrieval with a model, but keep the approved procedure visible as the output.
Include ownership in the comparison
Estimate the work needed to evaluate changes, maintain permissions, investigate errors, and keep source content current. Include the time a user spends reviewing or correcting answers. API cost alone is not the cost of the feature.
A model can still be the best choice. The useful outcome of this exercise is knowing which benefit justifies its maintenance burden and which risks the product must absorb. That makes a later architecture discussion much more concrete.
A decision record worth keeping
Write down the task, tested alternatives, representative failures, chosen approach, and the evidence that would reopen the decision. If the baseline wins, ship it without apologizing. If the LLM wins, keep the baseline as a comparison and, where appropriate, a fallback.
The durable product skill is not inserting a model into every workflow. It is identifying where uncertain language processing helps and where a deterministic path serves the user better. Continue with problem discovery and choosing the next layer of AI complexity.