Retrieval-augmented generation gives an application a way to bring external documents into the model's context. That is a useful capability, not a guarantee of correctness. The application can retrieve the wrong passage, retrieve an outdated policy, or use a correct passage to justify a claim it does not support.
The original RAG research establishes a retrieval-plus-generation approach for knowledge-intensive tasks. A production document assistant adds another set of obligations: source ownership, access control, versioning, and a response when the available evidence is insufficient.
Diagnose retrieval before rewriting the prompt
For each evaluation question, identify the evidence that should have been available. Check whether it was indexed, whether the user was allowed to access it, whether retrieval returned it, and whether the relevant text survived context assembly. A better generation prompt cannot repair a document that never entered the pipeline.
Test exact identifiers, abbreviations, tables, and questions whose answer spans adjacent passages. Structural chunking can preserve useful boundaries, but it is not automatically superior for every corpus. Compare chunking and retrieval choices on the same representative requests.
Authority is not the same as recency
Imagine two internal policies: a current policy approved last year and a draft uploaded yesterday. A blanket recency boost could promote the draft. The useful metadata includes effective dates, status, scope, and the relationship between versions—not just the upload timestamp.
When sources conflict, the system should use a documented precedence rule or expose the conflict. It should not silently merge incompatible statements into a smooth answer. Removing superseded content from default retrieval may be appropriate while retaining historical versions for explicitly historical questions.
A citation needs a support check
Distinguish citation presence, citation validity, and claim support. A link can exist and open successfully while pointing to a passage that says nothing about the claim. Evaluate those checks separately. For consequential answers, a reviewer may need to inspect the exact source passage.
Automatically deleting uncited sentences does not establish that the remaining answer is true. It can also remove a qualification and leave a misleading fragment. Treat evidence checks as part of an evaluation and recovery system, not as a proof-producing filter.
Handle different kinds of unanswered questions
A question about the future is not automatically unanswerable by retrieval: an approved schedule may state a future event. Conversely, a question about the past may be unanswerable because the records are missing. The useful distinction is whether the evidence supports the requested claim.
Separate lookup, synthesis, and prediction in the product contract. Synthesis across several documents can be a legitimate RAG task when coverage and attribution are evaluated. A novel forecast needs an explicit forecasting method and uncertainty treatment; it should not masquerade as a retrieved fact.
Keep permissions in the retrieval boundary
Apply the user's access rules before sensitive passages enter model context, and ensure caches do not cross that boundary. Asking the model not to reveal a document after giving it that document is not an authorization system. Deletions and permission changes also need a route into the index and caches.
RAG becomes useful when these responsibilities are visible and tested. Start with the architecture decision, then use task-based evaluations to decide whether retrieval actually improves the product.
Measure an evidence chain
A useful diagnostic separates document coverage, relevant evidence in the candidate set, evidence retained after reranking, faithful context assembly, and supported generation. Define relevance at the passage or document level before reporting recall. If several passages are needed for an answer, a single relevant hit may not establish sufficient coverage. Grounding tests should inspect the claim and its qualification, not simply whether a citation string appears.