RAG vs fine-tuning: which one your LLM project actually needs

"Should we fine-tune a model on our data?" is one of the first questions I hear on AI projects. Usually the honest answer is "probably not yet". Most teams asking it actually need retrieval, a better prompt, or both. Fine-tuning has its place, but it solves a different problem than most people think.

Here's how I decide between RAG and fine-tuning, what each one costs to build and maintain, and the cases where you need neither.

The short version

RAG (retrieval-augmented generation) gives the model the right information at the moment it answers. The system searches your documents, puts the relevant passages in the prompt, and the model answers from them.
Fine-tuning changes the model's behaviour. You train it further on examples, so it learns a format, a style, a vocabulary or a narrow task.

A useful rule: RAG is for knowledge, fine-tuning is for skills. If the problem is "the model doesn't know our prices, policies or products", that's knowledge. If the problem is "the model knows enough, but doesn't answer the way we need", that's a skill.

Start with neither: fix the prompt first

Before choosing, check whether a clear prompt already solves it. A surprising number of "we need fine-tuning" projects are fixed by:

precise instructions, with what to do when information is missing,
a few good examples of input and expected output in the prompt,
a defined output format, ideally structured (JSON with a schema) when another system reads it,
picking a model that suits the task.

This takes days, not weeks, and it gives you a baseline. Without a baseline you can't tell whether RAG or fine-tuning helped at all.

When RAG is the right choice

RAG fits when answers depend on information that is:

Specific to your business: product catalogue, contracts, internal procedures, past tickets.
Changing: prices, stock, policies. Update the documents and the next answer uses them. No retraining.
Something you need to cite: RAG can show which document an answer came from, so a person can check it.
Access-controlled: you can filter what each user is allowed to retrieve. A fine-tuned model can't "forget" a document for one user.

RAG also lowers the risk of the model inventing facts, because it answers from text you supplied. It doesn't remove that risk. The retrieval has to find the right passage, and the model has to stick to it. Most RAG failures I see are search failures, not model failures. I wrote a separate guide on building a RAG pipeline that holds up in production.

When to fine-tune an LLM

Fine-tuning earns its cost when you need the model to do something consistently that prompting can't get reliably:

A strict output format or style across thousands of calls: a house writing style, a specific report layout, a classification scheme with many labels.
A narrow, repetitive task where a small fine-tuned model can match a much larger general model. That matters for cost and for running the model on your own hardware.
Domain language the base model handles poorly: specialised jargon, or a dialect or language that is underrepresented in its training data.
Shorter prompts: behaviour that would need long instructions and many examples on every call can be baked into the model.

Today, fine-tuning usually means parameter-efficient fine-tuning, such as LoRA. Instead of updating all of the model's weights, you train a small set of extra weights on top of a frozen base model. It needs far less GPU memory, and the result is a small adapter file you can swap in and out. That makes fine-tuning open models on modest hardware realistic.

What fine-tuning is bad at: teaching facts that change. If you fine-tune prices into a model, you have to retrain every time they change, and you can't easily see where an answer came from.

What each one costs

RAG:

Building the ingestion and search pipeline: parsing documents, chunking, embeddings, a search index.
Ongoing work keeping the index fresh and permissions correct.
Longer prompts on every call, because retrieved passages are added to them.

Fine-tuning:

Collecting and cleaning training data. This is the real cost. You typically need hundreds to thousands of good examples, and they must reflect what you actually want, mistakes included.
Training runs and evaluation, repeated when the base model or your requirements change.
Hosting the fine-tuned model, or paying a provider's rate for custom models.

In both cases you need an evaluation set: a few hundred real examples with known good answers, so you can measure whether a change helped. Skip it and you're guessing.

When to use both

Some systems need both. A common pattern: fine-tune a model so it reliably follows your format and tone, then use RAG to feed it current facts. For example, a support assistant fine-tuned on how your best agents write replies, with answers grounded in your current policies. The fine-tuning handles how it answers, the retrieval handles what it says.

I only recommend this once RAG alone is measured and falls short. It's two moving parts to maintain instead of one.

A decision checklist

Does a good prompt with a few examples already work? Then stop there.
Is the problem missing or changing information? Use RAG.
Do you need citations or per-user access control? Use RAG.
Is the problem consistent format, style or a narrow task at high volume? Consider fine-tuning.
Do you have hundreds of high-quality examples? If not, you're not ready to fine-tune.
Do you want a small model on your own servers for cost or privacy? Fine-tuning a small open model is often how you get there.

The private-data angle

Both approaches can keep sensitive data in-house. RAG can run with a local model and a local search index, so documents never leave your network. Fine-tuning an open model on your own hardware means your training data never goes to a third party either. If you're weighing that, I compare the options in private AI and on-premise LLMs.

One caution: a fine-tuned model can reproduce pieces of its training data. Don't fine-tune on personal data you wouldn't want the model to repeat to the wrong user. RAG with proper access filtering is usually the safer choice for that kind of data.

Working together

I'm Driss Briksine, a software and ML engineer based in Morocco. I build AI systems end to end, from retrieval and model work to the production software around them, and I run most of my own systems on local models. You can see my case studies.

If you're deciding between RAG and fine-tuning, send me a short description of the task. I'll tell you which one I'd try first, and how to measure whether it worked.