How to measure if a RAG responds well: evaluation, sources, and difficult questions
Published on 2026-11-16 by ArduBot
Reproducible method to evaluate RAG with real questions, recall@k, citation accuracy, abstention, latency, and version changes.
“The chatbot answered me well three times” is not sufficient proof to offer RAG to clients. Quality must be measured separately in retrieval, generation, security, and maintenance. This way you can improve the correct component instead of changing models blindly.
Build a question set with verifiable answers
Select 50 real or representative user queries. For each one, store the question, correct document, expected section, acceptable answer, and whether the question should remain unanswered. Include: semantic questions, exact names, numerical references, contradictory documents, outdated information, and queries that the system should not know about.
Example: a manual states that equipment X requires 12 V, but an outdated version mentions 9 V. The question “What power supply does X require?” should retrieve the current version and show the reference. It is not enough for the model to say “12 V” by coincidence.
Two different measurements
Retrieval: Does the correct source appear in the first k fragments? You can use recall@k for a labeled battery and also review precision and result order. If the search engine does not deliver the correct clause, the model does not have reliable evidence.
Answer: Does it contain the permitted facts and nothing that contradicts the source? Does it quote the exact fragment? Does it abstain when evidence is missing? An automatic evaluator can help, but it is advisable to manually review a sample of errors, especially if there are economic or legal risks.
Metrics that should be on the dashboard
Record rate of questions with sufficient evidence, percentage of verifiable citations, errors per document, unanswered queries, p50 and p95 times, provider errors, and cost per query. If multi-company RAG is used, add client isolation tests.
Do not turn every metric into a global score: 95% retrieval can hide dangerous answers in exception questions. Indicators need concrete examples and error review.
How to detect the cause of a failure
If the correct fragment does not appear, inspect extraction, chunking, filters, and search. If it appears but the answer contradicts it, review the prompt, the model, and citation rules. If an old version appears, review synchronization. If foreign content is retrieved, the failure is authorization-related and requires maximum priority.
A change that improves general questions can worsen exact references. Always compare the same battery before and after, and keep document and configuration versions.
Example weekly cycle
Monday: extract ten failed questions. Tuesday: classify whether the failure is in extraction, retrieval, or generation. Wednesday: change one variable (for example, filters or chunking). Thursday: run the full battery. Friday: publish internal results, without glossing over unanswered cases.
What a buyer should validate
Do not ask the customer to believe a screenshot: let them test questions from their own documentation with expected answers. This transforms a generic demo into a concrete commercial evaluation.
ArduMaker is studying a future RAG API; its viability will depend on this measurable quality as well as commercial interest. If you would like to participate in a test, describe what documents and questions you would use at ArduMaker.
Sources: RAG Evaluators in Microsoft · Retrieval Quality in Databricks.