RAG — Retrieval Augmented Generation: Why LLMs Need Reference Books Before Answering

A few months ago, I was proud to have a local AI Assistant who could answer everyday questions. Ask about Nginx, he answers. Ask about Python, he answers. Ask about the recipe, he also answered. But over time I realized there was one annoying problem: the answers were sometimes too general, sometimes out of date, or even made up facts that I never actually gave. An LLM is like that friend who is very fluent, knows a lot of patterns by heart, but doesn't always have the right reference books with him. This is where I started learning about Retrieval Augmented Generation, or what is often abbreviated as RAG.

LLM Without RAG: Like a Friend Who Memorizes But Has No Notes

Just imagine that you are taking an exam. There are two friends sitting next to you. The first friend is very intelligent and knows many things by heart, but he only relies on memory. The second friend is not as nice as the first, but he has a neat notebook containing material from class, lecturer slides, and personal summaries. When the exam questions come out, the first friend can answer quickly, but sometimes gets the details wrong. The second friend answered a little slower, but more precisely because he matched the question with his notes first.

LLM without RAG is like a first friend. He was trained from billions of texts on the internet, so his knowledge is extensive but static. It doesn't know your latest documents, your personal notes, or your server configuration. RAG turns it into a second friend: before answering, it searches your personal database for relevant sources, then answers based on those sources.

What is RAG?

RAG is an AI architectural pattern that combines two things: retrieval (information search) and generation (text generation). In simple terms, the flow is like this:

1. User asks a question.
2. The system searches for the most relevant documents or pieces of text from the database that we have prepared.
3. The relevant documents are inserted into the LLM prompt as context.
4. LLM answers questions based on that context.

In this way, LLM's answers are no longer purely from his own memory. He now has a "reference book" that he can open before speaking.

Three Main Components of RAG

RAG implementation usually consists of three main components. Each has a different role, but supports each other.

1. Source Document

This could be a PDF file, a Markdown note, a documentation page, a server log, or even the blog article itself. The more quality and structured your document is, the better the RAG results will be. Garbage in, garbage out — that adage applies here too.

2. Vector Database

Vector databases store a numeric representation of text, called embedding. Imagine each sentence transformed into a set of numbers that capture its meaning. When users ask questions, the questions are also converted into similar numbers. The system then looks for documents whose numbers are closest to the question. Popular examples: ChromaDB, Qdrant, Milvus, and Weaviate.

3. LLM as Generator

Once the relevant documents are found, it is the LLM that compiles a human response. You can use local models like Qwen or Llama via Ollama, or use APIs like Gemini and OpenAI. The good thing is, RAG makes even a small LLM more useful because he doesn't need to memorize everything — just be smart about reading context.

Simple Implementation with Python and ChromaDB

To understand RAG, I prefer to immediately look at a small example. Here's a simple Python script that reads a few sentences, saves them to ChromaDB, and then answers questions based on that data.

# install dulu: pip install chromadb sentence-transformers
import chromadb
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction

# 1. Siapkan dokumen sumber
 documents = [
    "Nginx adalah web server yang populer dan ringan.",
    "Docker memungkinkan aplikasi berjalan dalam container yang terisolasi.",
    "RAG membantu LLM menjawab berdasarkan data pribadi pengguna.",
    "fail2ban melindungi server dari brute-force attack dengan memblokir IP."
]

# 2. Buat koleksi di ChromaDB
client = chromadb.Client()
ef = SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
collection = client.create_collection(name="catatan-server", embedding_function=ef)

# 3. Masukkan dokumen
collection.add(documents=documents, ids=["d1", "d2", "d3", "d4"])

# 4. Tanya dan ambil konteks relevan
query = "Cara melindungi server dari serangan password?"
results = collection.query(query_texts=[query], n_results=2)
context = "\n".join(results["documents"][0])

print("Konteks yang ditemukan:")
print(context)

If the script above is executed, ChromaDB will return the documents most relevant to the question. Documents about fail2ban and Docker are likely to appear, as they both deal with server security. The search results can then be compiled into a prompt like this for LLM:

Kamu adalah asisten teknis. Jawab pertanyaan berikut berdasarkan konteks di bawah.

Konteks:
- fail2ban melindungi server dari brute-force attack dengan memblokir IP.
- Docker memungkinkan aplikasi berjalan dalam container yang terisolasi.

Pertanyaan: Cara melindungi server dari serangan password?

LLM will answer with more focus, because it has a specific context. He no longer guesses or gives general answers from his old knowledge.

When is RAG Really Needed?

RAG is not a solution to all problems. There are situations where RAG is very useful, and there are situations where it makes the system more complicated for no apparent benefit.

RAG is suitable for use when you have personal data that cannot be provided to LLM directly, such as server records, internal documents, or blog archives. RAG is also useful when you want LLM answers to always refer to a specific source, for example the official documentation of the product you are developing. In addition, RAG helps reduce hallucination, which is the tendency for LLMs to make answers that sound convincing but are wrong.

However, RAG is not really needed if the user's question is general and does not require specific data. Ask about world history or writing tips, LLM without RAG is enough. Ask about the server configuration you just changed last week, then RAG becomes important.

Limitations to Keep in Mind

RAG sounds perfect, but it actually has limitations. First, the quality of the answer is very dependent on the quality of the source document. If the document is messy, the search results will also be messy. Second, RAG does not always find the most appropriate context, especially if the question is phrased with different keywords than the content of the document. Third, RAG adds complexity: you need to think about how to crop documents, save embeddings, update data, and manage vector databases.

So RAG is like building a personal library. It's not enough just to have a bookshelf, but you also have to have a neat catalog system, staff who know where the books are, and the habit of tidying them up after use.

Conclusion

RAG opens up a new way to utilize LLM: from just a machine that memorizes patterns, to an assistant that can read and refer to our own data. For me who has a home server and lots of configuration notes, RAG is like giving the AI ​​Assistant a personal reference book. He doesn't need to know everything, as long as he knows where to look.

If you already have a local LLM or your own AI assistant, trying RAG is a pretty fun next step. Starting from a small document, one collection, one question. Over time, you will have a digital library that you can chat with.