Before You Build a RAG Assistant, Test These Five Things
A useful RAG assistant needs more than connected documents. Start with a clear task, current sources, reliable retrieval, access controls, and a way to check its answers.
- Choose one question the assistant should answer
- Check the documents before the model
- Test whether search finds the evidence
- Keep document access intact
- Write down what a good answer requires
- Use failures to choose the next change
Choose one question the assistant should answer
Retrieval augmented generation, usually shortened to RAG, combines search with a language model. The system finds relevant information and includes it in the context used to generate an answer. This can help an assistant work with business documents, but connecting a folder does not establish that the answers are useful.
Start with one task. In a fictional company, a travel assistant might explain reimbursement rules to employees. Define the documents it can use, the questions it should cover, and when it should direct someone to a person.
Check the documents before the model
Review the material as if a new colleague had to use it. Can they tell which policy is current? Are tables readable? Do headings describe the sections beneath them? Remove obsolete copies from the active collection or clearly mark their status.
Give each document an owner and a review date. Preserve its title and source location when processing it into smaller searchable sections. A passage separated from its heading or footnote may lose the context needed to interpret it correctly.
Test whether search finds the evidence
Retrieval and answer generation are separate parts of the system. Inspect the passages returned for a question before judging the final response. If the relevant rule never reaches the model, rewriting the answer instructions may leave the underlying problem unresolved.
For the fictional travel assistant, try both "Can I claim a taxi?" and "Is local transport reimbursable?" Check whether each retrieves the appropriate rule. Also try a question about an exception. Record missing evidence and irrelevant passages instead of relying on how confident the answer sounds.
Keep document access intact
A searchable document is not automatically appropriate for every user. Map the assistant's access to the permissions that govern the original information. Apply authorisation in the retrieval system, with the user's identity available where needed.
Test with accounts that have different permissions. An employee asking about travel rules should not receive restricted payroll details merely because those files share a storage location. Include this boundary in the pilot review, alongside answer quality.
Write down what a good answer requires
Create an evaluation sheet with the question, expected source, required facts, retrieved passages, actual answer, and review result. Check whether cited material supports the answer, rather than treating the presence of a citation as proof.
Include ordinary questions, unclear requests, outdated information, inaccessible documents, and questions the collection cannot answer. For missing evidence, define an acceptable response that acknowledges the gap and offers a useful next step. Measure response time alongside quality.
Use failures to choose the next change
Group failures by cause: missing content, poor extraction, retrieval mistakes, permission errors, or unsupported answers. Fix the relevant part and rerun the affected questions. Expand the collection only after the pilot gives a clear view of these failure modes.
A practical first deliverable is a reviewed question set and an issue log. Together, they make the decision to continue, narrow the scope, or change the design easier to explain.
Further reading
Primary references behind the technical guidance in this article.
Put the thinking into practice.
Explore our generative ai & llms services, or talk to us about the task you want to improve.
Explore the service
