RAG, Fine-Tuning, or Neither? Start with the Failure
“Should we use RAG or fine-tune?” sounds like an architecture question. In practice, it is several diagnostic questions compressed into one. Is the system missing information? Is it applying the wrong behavior? Is the task constrained enough that generation is unnecessary?
I would begin with a small set of failed tasks and label what went wrong. Selecting a technique before doing that can produce a more elaborate system with the same underlying defect.
When the evidence is missing
Suppose a hypothetical support assistant answers from an old product description because the current documentation is not in its context. Retrieval is a candidate intervention: locate the right material and supply it at answer time. The evaluation then needs to test document coverage, permissions, passage relevance, and faithful use of the evidence.
Do not jump directly to a vector database. Exact identifiers may suit structured lookup or keyword search. A small, bounded reference set may fit directly in context. Compare approaches with the real information need rather than treating one retrieval implementation as the definition of RAG.
When the behavior is inconsistent
A different failure occurs when the model has the necessary information but repeatedly uses the wrong format, ignores a task boundary, or applies a classification policy inconsistently. Clarifying the instructions, adding representative examples, or moving constraints into code may be the first things to test.
Fine-tuning becomes a candidate when there is a repeatable behavior to learn, suitable training examples, and an evaluation that can distinguish real improvement from memorization. It brings a training-data lifecycle and a new versioned artifact. It is not a shortcut around defining the task.
Knowledge and behavior are not a clean wall
It is useful to distinguish access to evidence from learned behavior, but the techniques can interact. A fine-tuned model can still use retrieval, and retrieval quality can depend on model behavior. The original RAG work itself combines retrieval with trained generation rather than presenting a universal either-or choice.
For a product team, the important question is which change addresses the measured failure with the least additional burden. A combination may be justified, but it should emerge from evidence rather than from a checklist of fashionable components.
Sometimes the answer is a form or a rule
If the task is checking whether a required field is present or fetching a known record, a deterministic path may be easier to test and explain. A model can help interpret a messy request at the boundary without becoming the authority for the underlying business rule.
For example, a system might extract a product identifier from free text, validate it against a catalogue, and return the catalogue's availability. It should not ask a model to invent an availability estimate when the lookup fails.
Run an intervention-sized comparison
- Keep the task set and success criteria stable.
- Change one important layer at a time where feasible.
- Compare against the current workflow, not only another model.
- Inspect new failure types introduced by the intervention.
- Include latency, review effort, and maintenance in the decision.
Record what would invalidate the choice: changing source material, a new language, a different permission model, or a workload outside the training examples. An architecture decision is a response to a particular task and operating context, not a permanent declaration of allegiance.
Read the remaining failure modes in retrieval systems and how to build a useful non-LLM baseline before adding another layer.