RAG or Fine-Tuning? How to Make an AI Assistant Answer From Your Documents

Sooner or later every company that tries a general AI chatbot asks the same question: how do we make it answer from our own documents — our policies, manuals, contracts and past tickets — instead of from the internet? Two techniques come up in every answer: retrieval-augmented generation, usually shortened to RAG, and fine-tuning.

They are often presented as alternatives, but they solve different problems. Choosing the wrong one is one of the most common reasons company AI assistants disappoint.

What fine-tuning does

Fine-tuning continues training a language model on your own examples, so it adjusts how the model behaves: the tone it writes in, the format of its answers, the kind of task it expects. It is useful when you need consistent behaviour that is hard to describe in instructions — a particular writing style, a structured output, or a narrow classification task.

What fine-tuning does not do well is give the model reliable knowledge of facts that change. Facts learned during training cannot easily be updated or removed, the model cannot show where an answer came from, and if a document changes next week, the model is already out of date.

What RAG does

RAG leaves the model as it is and gives it the right information at the moment a question is asked. When someone asks something, the system first searches your documents for the most relevant passages, then asks the model to answer using only those passages, and shows which documents it used.

Because the knowledge lives in the search index rather than in the model, updating it is as simple as updating the documents. Answers can cite their sources, access rules can be applied during the search, and the same model can serve many teams with different document sets.

Which one do you need?

If the goal is “answer questions from our documents”, start with RAG. That covers internal knowledge assistants, support bots that answer from help-centre articles, and tools that search contracts or specifications.

Consider fine-tuning when the problem is behaviour rather than knowledge: the output must follow a strict format, match a specific tone, or perform a repetitive task where good instructions and a few examples are not enough. The two combine well — a fine-tuned model can still answer from retrieved documents — but most companies get the bulk of the value from a well-built RAG system alone.

What makes a RAG assistant accurate

Most disappointing RAG assistants fail at the search step, not the language model. If the wrong passage comes back, the model answers the wrong question confidently. Good retrieval splits documents along their real structure, combines meaning-based and keyword search so product codes and names are found exactly, and re-ranks results before the model sees them.

The second ingredient is honesty. The assistant should answer only from what it retrieved, link every answer to its source, and say “I don’t know” when the documents do not cover the question. The third is measurement: a test set of real questions with correct answers, scored before launch and after every change, so quality is known rather than assumed.

Permissions and privacy

A company assistant must not become a way around access rules. Each document should carry its permissions into the search index, and every search should be filtered by who is asking, so nobody receives an answer drawn from a file they could not open themselves.

Where the documents may be processed matters too. Depending on your requirements, the model can be a hosted service under terms that exclude training on your data, hosted in an EU region for GDPR, or an open model running entirely inside your own cloud.

A practical way to start

Pick one well-defined document collection and one group of users — a support team and its help-centre articles is a classic first case. Collect fifty to a hundred real questions with their correct answers, build a prototype, and measure how often it finds the right passage and answers faithfully. Expand to more documents and more teams only when those numbers hold.

This is how we build RAG systems at MLMind: retrieval first, a citation on every answer, permissions built in, and accuracy measured on your own questions before anyone relies on it.