Yes, retrieval-augmented generation and fine-tuning can be combined, and in mature systems they often are — retrieval supplies current, citable facts while a fine-tuned model handles output format, house style or specialised vocabulary. But they solve different problems, so combining them before you can point at a specific failure that fine-tuning fixes adds cost and a retraining cycle for no gain. Start with retrieval, measure, and fine-tune only against a named deficiency.
The question gets asked as though it's a fork in the road. It isn't. Retrieval and fine-tuning address different failures, and most teams asking which to choose have only one of those failures.
The one-line distinction
Retrieval changes what the model knows at the moment you ask. Fine-tuning changes how the model behaves in general. If your problem is that the model doesn't know something, retrieval. If your problem is that it knows but answers wrongly-shaped, fine-tuning.
Which failure do you actually have?
| Symptom | What fixes it |
|---|---|
| Doesn't know your products, policies or customers | Retrieval. The facts aren't in the model and shouldn't be. |
| Answers go stale when your data changes | Retrieval. Fine-tuning bakes knowledge in at training time; it is stale the day after. |
| Can't cite where an answer came from | Retrieval. A fine-tuned model has no source to point at. |
| Right facts, wrong format or tone, every time | Fine-tuning. Or better, a stricter output schema first — try that before training anything. |
| Handles your domain's vocabulary clumsily | Fine-tuning, once you've confirmed a better prompt and a glossary in context don't fix it. |
| Too slow or expensive at your volume | Neither. That's a routing, caching and context-trimming problem. |
The disadvantages of RAG, stated plainly
Retrieval is our default recommendation and it has real costs that vendor pages skip.
- It adds latency. You search before you generate. With re-ranking, several hundred milliseconds before the model starts. Streaming hides it; it doesn't remove it.
- It adds per-query cost. Retrieved passages are input tokens, every time. Pass too many and generation cost doubles for no accuracy gain.
- It is capped by your content. RAG over stale documentation produces confident, well-cited, wrong answers — and the citation makes them *more* persuasive. If your corpus is a mess, fixing the corpus is the higher-return project and no retrieval system will substitute for it.
- Retrieval quality is invisible without measurement. When the right passage isn't found, the model still writes a fluent paragraph from the wrong ones. Nothing errors.
- It's more infrastructure. Ingestion, indexing, permission sync, staleness monitoring. All of it needs owning after launch.
The disadvantages of fine-tuning, stated plainly
- Knowledge goes stale immediately and updating means retraining, not editing a row.
- No citations. The model cannot tell you where it learned something, which rules it out anywhere an answer must be verifiable.
- You need training data — typically a few hundred to a few thousand good examples. Producing them is usually the real cost, and it's human time, not compute.
- It's a versioned artefact. Every base model upgrade means retraining and re-validating. You've taken on a maintenance obligation that renews on the provider's schedule.
- Evaluation gets harder, because a regression can now come from the data, the training run or the base model.
So when does combining them make sense?
When you have measured a specific deficiency that retrieval structurally cannot fix. Three genuine cases:
- 01A rigid output format the model keeps deviating from. You've tried a schema and a stricter prompt, and it still drifts on 5% of calls at volume where 5% matters.
- 02A house voice that matters commercially. Legal, medical or regulated writing where the phrasing is part of the product and prompt-based style instructions are inconsistent.
- 03Domain vocabulary the base model mishandles. Genuine specialist jargon — not just industry terms, which retrieval and a glossary handle fine.
In all three, the architecture is: retrieval supplies the facts, the fine-tuned model shapes the answer. They compose cleanly, because they're operating on different things.
What each costs to build
| Retrieval | Fine-tuning | Both | |
|---|---|---|---|
| Build time | 4–8 weeks | 3–6 weeks | 8–14 weeks |
| Main cost driver | Ingestion + retrieval tuning | Producing training data | Both |
| Running cost | Per query, context-dependent | Lower per token, hosting if self-served | Per query |
| Updating facts | Re-index — minutes | Retrain — days | Re-index |
| Citations | Yes | No | Yes |
| Maintenance burden | Ingestion health, index freshness | Retrain on base model upgrades | Both |
Is there anything better than RAG?
For grounding a model in your own changing content, no — not as a general replacement. But two things frequently beat it for a specific problem, and both get overlooked because they're less interesting:
- Just putting the content in the prompt. If the relevant material fits comfortably in the context window — a single policy document, one customer's account history — retrieval is machinery solving a problem you don't have. Long context windows made this viable for far more cases than it used to be.
- A structured query instead of a search. If the answer lives in your database and the real difficulty is that users can't express the query, the right build is often text-to-query over your schema rather than retrieval over documents. It's more accurate, cheaper, and verifiable — the query itself is the citation.
Both are worth ruling out before committing to an ingestion pipeline. We do exactly that during discovery, and it occasionally ends with us recommending the cheaper thing. How we build retrieval systems covers what happens when retrieval genuinely is the answer.
The short version
Start with retrieval, because it's cheaper to build, far cheaper to change, and fixes the problem most teams actually have. Add fine-tuning only when you can name the failure it addresses and show it in a test set. And before either, check whether the content simply fits in the prompt, or whether the question is really a database query wearing a search box.