RAG Systems, Explained: Why it is not Just a Smart Filing System

A Retrieval Augmented Generation system is a technique built on top of an AI engine. Its job is to find the relevant material inside a collection of documents you have supplied and, depending on how it has been set up, write a grounded answer from it.

Why Would Someone Need a RAG System?

Having a RAG system is like having an AI engine at your disposal that answers only from a limited document corpus you provide. The model doing the writing is no less capable than any other; the pool it draws on is simply narrower, and entirely yours.

Sometimes there are sensitive company documents that cannot be uploaded to a third party’s servers at all, which rules out the public AI engines entirely. There are times when the sheer vastness of AI engine’s knowledge repository drowns out the sharpness of your subject.

A RAG system works from a smaller, custom collection, holding only the information and documentation you submit to it. Depending on how it is built, the whole thing can sit on a private server running on the intranet of an individual or company, so no document ever leaves the building.

How Exactly Does A RAG System Work?

Say you are an NGO, specialising in girl-child education with offices across the country. This means there are varying statistics and pan-India documents to be categorised and referred to. Instead of each office having their own library of files and having to corroborate information with another office before putting out statements, a RAG system puts all the required information in one place.

All internal documents on an intelligent repository, depending on the particular RAG’s capabilities, can be spoken to like any other chat engine and the information will be sourced from the specific document and cited. An example of the kind of response to expect would be this.

“Please source the 2016 press release on the Odisha Beti Bachao, Beti Padhao initiative we helped promote alongside the Ministry of Women and Child Development. On the basis of that information, create a statement in conjunction to the 2026 report on the initiative from April.”

Depending on how the RAG is built, and whether a language model such as Llama, Qwen or DeepSeek sits alongside the Python retrieval script, it may write answers rather than simply hand back documents. That is what turns it into something you can ask for a deliverable. The two parts are worth separating: the Python script is the code that does the searching and the prompting, while the model is an enormous file of trained numbers that the script calls on to produce language.

Ollama is a piece of software that runs a language model on your own machine. The model simply carries everything it learned during training, and can draw on that while writing. So there is no live crawl and no scheduled refresh, but there is a layer of general knowledge sitting behind the answer that goes beyond the documents you submitted. This is true of any language model, cloud or local; Ollama’s only contribution is keeping it on your machine.

\~Analogy break\~

Think of this whole situation in an office setting

Your LLM would be the consultant. They are expensive to train, have extensive knowledge about their domain but, can also be changed tomorrow.

Ollama is the office through which the consultant works. It has a whole system in place, a cabin, a desk, a phone line, to allow your consultant to work. Even if the consultant changes tomorrow, the office remains the same.

Your RAG System is the research assistant. Their main job is to retrieve files, categorise the relevant material and put it on the consultant’s desk so they can finish the work for the client i.e., you. The consultant has never seen the assistant’s filing system and therefore, only knows what is being given to them, grounding the answer in their own experience and the files presented.

When the RAG has no language model attached to it, it will simply retrieve the relevant passages from the collection and show them to you. The chunking happened earlier, when the documents were first loaded in. By the time you ask a question, everything is already organised in categories and waiting.

In the case of the above example,

1. The system would retrieve the documents mentioned itself or,
2. Retrieve the relevant passages from within those documents or,
3. Create the asked deliverable

The Difference Between A RAG System and A Generative AI Engine

Both take a question and return with a sourced answer. The difference is in the source of the documents they read.

A RAG system searches a corpus you supply. You choose what goes in, so you control what comes out. It only knows your files, and it knows nothing else about the world.

An AI generative engine, whether ChatGPT, Perplexity, Gemini or AI Overviews, searches the open web. Nobody controls the index. What surfaces is whatever the retrieval layer judges relevant from everything published, and the brand being discussed has no direct say in it.

Building that private, cited retrieval layer for a client is the work we do under Context Engineering.

Sources

• Lewis, P. et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 33 (2020): foundational RAG paper, backs the definition of RAG as a technique built on top of a pre-trained model rather than the foundation beneath it. arxiv.org/abs/2005.11401
• Gao, Y. et al., “Retrieval-Augmented Generation for Large Language Models: A Survey,” arXiv:2312.10997 (Dec 2023, rev. Mar 2024): standard overview of the field, backs the retrieval, generation and augmentation breakdown and the point that RAG exists to address hallucination and outdated knowledge. arxiv.org/abs/2312.10997
• Ollama, project repository and model library: backs the description of Ollama as software that runs a language model on your own machine, and the separation between the retrieval script and the model it calls. github.com/ollama/ollama and ollama.com/library
• llama.cpp (ggml-org), project repository: the inference engine Ollama is built on, backs the point that Ollama is a delivery layer rather than an AI system in its own right. github.com/ggml-org/llama.cpp