RAG for Small AI Applications - Retrieve Evidence Without Treating It as Truth

When a language model answers a question about a private document, a product manual, or a changing internal policy, there is an uncomfortable gap between what the model knows and what the application needs it to know. Retrieval-augmented generation (RAG) is one way to narrow that gap. But retrieval is not a truth machine. It is a way to place selected material in front of a model, with all the risks that selection and interpretation bring.

This distinction matters for a small AI application. It is tempting to start with a vector database and promise answers grounded in company documents. A more useful starting point is a set of questions: which documents are allowed to be used, how will relevant passages be found, what happens when nothing relevant is found, and how will anyone know when the system is wrong?

What RAG Adds to a Language Model

A language model has knowledge represented in its parameters. RAG adds an external collection of material that can be searched at request time. The original RAG paper by Lewis and colleagues describes a model combining parametric memory with non-parametric memory: a retriever accesses a dense vector index, and the generator uses the retrieved material while producing an answer.

In a small application, the external collection might be a set of help pages, a database export, or carefully selected Markdown files. The model does not magically absorb these files as permanent knowledge. The application retrieves some passages for a particular question and includes them in the model input.

That makes RAG different from fine-tuning. Fine-tuning changes model behavior or parameters through additional training. RAG normally leaves the model parameters alone and changes the context supplied at inference time. The two approaches can be combined, but they solve different problems.

The Pipeline Has More Than a Vector Search

Documentation often presents RAG as a neat chain, but each stage can fail independently. A practical mental model has five stages.

1. Load the source material

First decide which files, pages, records, or API responses are part of the corpus. Loading is not a clerical detail. If an old policy, a draft, and the current policy all enter the corpus without metadata, retrieval may find the wrong version while still looking plausible.

Access rules belong here too. A retrieved passage should not reveal a document that the requesting user was never permitted to read. Filtering only after generation is too late: the model has already seen the restricted context.

2. Split and index the material

Long documents are usually divided into chunks. Each chunk can be represented by an embedding, a numerical representation intended to capture aspects of its meaning. The embeddings are stored in an index so that a query can find nearby material.

There is no universally correct chunk size. A chunk that is too small may lose the condition that gives a sentence its meaning. A chunk that is too large may contain several unrelated subjects and consume more context. Headings, document identifiers, dates, access labels, and links are often as important as the text itself.

The Dense Passage Retrieval paper describes a dual-encoder approach for learning dense representations of questions and passages. It is useful background for understanding why embeddings can retrieve semantically related passages, but its reported benchmark results are not a guarantee for a different corpus, language, model, or application.

3. Store the index and metadata

The index is only one part of the stored state. A useful result should still point back to a source document and its location. Store enough metadata to apply authorization, filter by version or date, and show a meaningful citation. Without that connection, a system may produce fluent text while making it difficult to inspect the material that influenced the answer.

4. Retrieve for a question

At query time, the application turns the question into a search request. Dense retrieval can find related wording even when few keywords overlap. The OpenAI Retrieval documentation, for example, describes semantic search over vector stores, result limits, scores, metadata filters, and optional query rewriting.

Keyword search is not obsolete. Product names, error codes, version numbers, and exact identifiers can be poorly served by semantic similarity alone. A hybrid strategy can combine lexical matching with embeddings, and a reranker can reorder candidates. These are design choices, not proof that a result is correct.

5. Generate and evaluate

The selected passages and the original question become context for the language model. The model then writes an answer. This is the generation step, not a database query returning a verified fact. The model can misunderstand a passage, combine two incompatible passages, or answer beyond what the context supports.

LlamaIndex describes loading, indexing, storing, querying, and evaluation as distinct stages. That separation is valuable because “the answer sounded good” is not an evaluation method. A small test set should include ordinary questions, questions whose answer is absent, ambiguous questions, outdated-document cases, and access-control cases.

Three Different Ways RAG Can Be Wrong

The retriever found the wrong passage

A search result can be topically related but fail to answer the question. For example, a document may describe how to rotate a credential while the question asks whether a particular user is allowed to rotate it. The words are related; the authorization rule is different.

The corpus does not contain the answer

Retrieval cannot recover information that was never loaded or that was removed during preprocessing. A responsible application needs an abstention path: “I could not find support in the supplied documents” is more honest than filling the gap from the model's general memory.

The generator overstates the evidence

Even relevant passages may be conditional, contradictory, or incomplete. Asking the model to include source identifiers can improve inspectability, but a citation is not automatically evidence that the sentence is entailed by that source. Citation display and factual verification are separate concerns.

A Small, Defensible Design

For a small application, the first version can be deliberately modest. Keep a versioned corpus, attach a stable identifier and access label to every chunk, retrieve a limited set of candidates, and pass their identifiers alongside the text. Tell the model to answer only from the supplied context and to state when the context is insufficient. Then log the question, retrieved identifiers, model answer, and reviewer judgment with appropriate privacy controls.

This is not a security boundary by itself. Untrusted documents can contain instructions aimed at the model, sensitive data can be retrieved for the wrong user, and logs can become a second source of leakage. The earlier article on prompt injection covered why instructions are not a security boundary; RAG adds another input path that must be treated with the same caution.

Evaluation should measure at least retrieval relevance and answer quality separately. Ask whether the expected source was retrieved, whether the response is supported by the retrieved text, whether it abstains when evidence is missing, and whether permissions were respected. NIST's AI Risk Management Framework is broader than RAG, but its emphasis on managing and evaluating AI risk is a useful reminder that one component does not make an entire system trustworthy.

What RAG Does Not Promise

• RAG does not make an outdated corpus current.
• RAG does not guarantee that the top result is relevant.
• RAG does not turn an untrusted document into a trusted instruction.
• RAG does not remove the need for access control, testing, or human review for consequential decisions.
• RAG does not make a model's answer equivalent to a quotation from its sources.

Conclusion

RAG is best understood as a retrieval-and-generation pipeline, not as a switch that turns a language model into a database. Its useful contribution is giving the model selected external context at the moment a question is asked. Its hard work is deciding what belongs in the corpus, what should be retrieved, who may see it, how missing evidence is handled, and how failures are measured.

For a small AI application, that modest framing is a strength. Before comparing vector databases, make the evidence boundary explicit and build tests around the failures you are willing to discover. The remaining question is not merely whether the model can answer, but whether the application can explain why that answer was allowed.

References

1. Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv, revised April 12, 2021.
2. Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering, Association for Computational Linguistics, EMNLP 2020.
3. OpenAI, Retrieval, official documentation, publication date not stated.
4. LlamaIndex, Introduction to RAG, official documentation, publication date not stated.
5. National Institute of Standards and Technology, AI Risk Management Framework, released January 26, 2023.