Compare · 01
As of September 2026
Build · AI Development · AI Implementation · Testing & Evaluation
Look it up,
or learn it
by heart?
Retrieval-augmented generation. At question time the system searches your documents and the model answers from what it found.
Further training on your examples. The model’s weights change, so the behaviour sticks without a lookup.
01 · Side by side
One changes the input.
One the model.
Both make a general model useful for your case. They work on different layers, fail differently and cost differently. Costs are relative: real numbers depend on model, volume and vendor.
RAGat question time, every time
- Question
- Search your index
- Question + passages
- Model
- Answer with source
Fine-tuningbefore, once per version
- Examples
- Training run
- Tuned model
- Question
- Answer
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What changes | The prompt: relevant passages are added at question time | The model: its weights are adjusted on your examples |
| Best at | Facts, documents, anything that must be current or cited | Format, tone, style, narrow tasks, consistent structure |
| Freshness | Update the index, and the next answer knows. Minutes to hours. | Frozen at training time. New facts need a new training run. |
| Data you need | Your documents: cleaned, split into chunks, with access rights attached | Curated pairs of input and ideal output for the behaviour you want. Quality beats volume. |
| Cost shape | Lower upfront, higher per query: a pipeline and an index first, then longer prompts on every call | Higher upfront, lower per query if a smaller model or a shorter prompt now does the job. Every retraining costs again. |
| Latency | Adds a retrieval step and a longer context | No lookup. Can be faster, especially with a smaller tuned model |
| Traceability | Answers can cite the passage they came from | No source to point at. The knowledge sits in the weights. |
| Access control | Filter the search by the user’s permissions, on every query | Whatever went into training is available to everyone who uses the model |
| Evals | Two layers: did it find the right passage, and did the answer use it faithfully? | Every trained version against the base model, on held-out cases, before it ships |
| Typical failure | The wrong passage is retrieved, and the answer is confidently built on it | Confident answers from half-learned facts |
02 · When to pick which
Pick by the
problem.
Concrete situations, and which of the two we would reach for first.
Pick RAG when
- 01Support answers from a product manual that changes every release
Freshness and citations. Retraining every release is the expensive way to be late.
- 02Employees ask about HR policies, and every answer must show the paragraph
Traceability is built into retrieval. Weights can’t quote.
- 03Different teams may see different documents
The search can filter by permission. A trained model can’t forget for one user.
- 04You have documents, but no labelled examples
RAG starts with what exists today.
Pick Fine-tuning when
- 01Every output must follow a strict structure, like a claims summary your team reads in ten seconds
Behaviour, not knowledge. Tune when prompting alone keeps drifting.
- 02Sorting a high volume of tickets into your own categories
A small tuned model can be faster and cheaper per call than a large prompted one.
- 03House style or domain shorthand must come out right every time
When the prompt fills up with examples, move the examples into the weights.
- 04It has to run on a small model, on your own hardware
Tuning can close part of the gap to a large model, on one narrow task.
And Before either: a good prompt with a few examples, and an eval set of real questions. Many projects never need more, and both approaches need the eval set anyway.
03 · Decide
Five questions,
one lean.
The lean moves while you answer. The recommendation appears when all five are in.
01What goes wrong today?
Knowledge or behaviour. The most useful question on this page.
02How often does the information change?
03Must answers show their source?
04Do different users see different data?
05What do you have today?
04 · Together
Better together,
in that order.
In production it is rarely either-or. Five places where the two meet.
- Facts from retrieval, format from tuningRAGBrings current passages into the promptFine-tuningTeaches layout, tone and when to say “not in the sources”common
- Tuning for retrievalRAGRetrieved passages include distractors that don’t answer the questionFine-tuningTrain the model to use the relevant passage and ignore the rest (RAFT, Zhang et al., 2024)research
- Better searchRAGSearch quality depends on the embedding modelFine-tuningEmbedding models can be tuned on your own pairs of question and documentcommon
- Smaller model, same jobRAGKeeps knowledge outside, so the model can stay smallFine-tuningDistils the behaviour of a large model into a smaller one, for speed and costcommon
- One eval set for bothRAGRetrieval hit rate and faithfulness of the answerFine-tuningBase model against tuned model, same questionsalways
Always: needed in every setup. Common: what we see in production. Research: published and promising, check it on your own data.
05 · Questions
Asked in every
architecture review.
Can fine-tuning teach the model our facts?
Partly, and poorly. A study comparing both found retrieval consistently ahead for injecting knowledge (Ovadia et al., 2023). Another found that training on new facts goes slowly and, once learned, raises the tendency to hallucinate (Gekhman et al., 2024). Use tuning for behaviour; keep facts where they can be looked up.
How many examples does fine-tuning need?
Fewer than people fear, better than people bring. One vendor’s guide names a minimum of 10 and recommends starting with 50 well-crafted examples, then evaluating (OpenAI, checked September 2026). The same guide says to invest in fine-tuning only after setting up evals.
Doesn’t a long context window make RAG unnecessary?
Not usually. Putting everything into the prompt costs more per call, gets slower, and makes it harder to control who sees what. Retrieval is also a filter.
Is our data safer with one of them?
With RAG, documents stay in your store and can be filtered per user. With fine-tuning, whatever went into training can come out for any user of that model. Personal data in a training set needs a legal basis and a plan for deletion requests: talk to your data protection officer before, not after.
Can every model be fine-tuned?
No. Many closed models offer no fine-tuning, or only for some versions or on some clouds. Open-weight models can be tuned wherever you run them, for example with LoRA, which trains a small set of added weights instead of all of them (Hu et al., 2021).
What do we need before either?
An eval set: real questions with expected answers, including the ugly ones. Without it you can’t tell whether retrieval or tuning helped, or whether the next change broke something.
Still torn?
Measure,
then pick.
Bring real questions. The eval set comes first, so the choice is a number, not an opinion.