The first question on most applied LLM projects is "should we use RAG or fine-tune the model?" The two get talked about like rivals, but each one solves a different half of the problem. Retrieval-augmented generation (RAG) feeds the model relevant documents at query time. Fine-tuning runs extra training on your own examples so the model learns a behavior. Once you separate those jobs, the choice is mostly about where the knowledge lives and how often it changes.
What is each one actually for?
RAG is for answers that depend on facts outside the model: policies, product documentation, customer records, anything that will change after the model was trained. You index the material, pull the relevant pieces at query time, and the model answers from what it was given.
Fine-tuning is for behavior: output format, tone, structure, the way the model should classify or extract in your domain. You show it enough examples of the behavior you want and it generalizes from there.
When does retrieval win?
When the facts change. Policies get rewritten, prices move, documentation gets updated, a new customer brings a new data shape. RAG wins because updating the answer means updating the index, not retraining the model. A document edit is a deploy. A training run is a project.
Retrieval also wins when you need citations. Retrieved passages give you sources to cite, but you still need to check that each citation supports the claim. Retrieval alone does not guarantee a grounded answer.
When does fine-tuning win?
When the behavior is stable and the output shape matters more than the facts. A fixed note format, a consistent tone, a classification scheme that will not change next quarter, a structure the model keeps missing no matter how the prompt is worded. Fine-tuning is the dependable way to make the model do a thing you can describe with examples.
It also wins when latency and per-call cost matter. A tuned model can be small and fast, because it does not drag a pile of retrieved context along on every request.
How do the costs compare?
RAG is cheap to start and pays per query: embeddings, retrieval, and the tokens for the context on every request. The bill scales with traffic and with how much context each request needs.
Fine-tuning adds up-front training and evaluation costs; per-call savings depend on the model and workload. A smaller tuned model can lower serving costs, but a model upgrade may require another training and evaluation cycle.
The honest comparison is not the sticker price. It is cost per good answer at your expected volume, including the human review time each approach needs when it fails.
How should you test the choice?
For example, consider a support assistant answering refund questions. Keep the current refund policy in retrieval so a policy edit can change tomorrow's answers. If the response must follow a fixed classification schema, test prompting first, then fine-tuning if it still fails. An evaluation case should change the policy while keeping the required output format unchanged.
Build the eval set before you argue about architecture. Collect the cases that actually show up: the odd request, the out-of-date policy, the record that changed since it was indexed. Run both approaches against the same cases and count good answers, bad answers, cost, and latency. The numbers usually settle the debate faster than the meeting does.
Teams that skip this tend to switch back and forth, because each approach has a failure mode that looks fixable by the other. Retrieval fails when the right document is missing or stale. Fine-tuning fails when the behavior was never stable enough to learn. Both failures show up in a decent eval set.
Most of the work I review splits the same way: the facts go in the knowledge base, the behavior goes in the model. When the business changes a policy, you edit a document and reindex instead of retraining. When the format drifts, you retrain instead of rewriting every prompt. Keep the two jobs separate and each one stays cheap to change.
If this is the kind of problem your team is working through, I am always glad to compare notes.