When a company needs to put its own knowledge into a language model —its catalogue, its manuals, its ticket history— the same argument almost always comes up: RAG, fine-tuning, or just stuff it all into the context? The conversation usually revolves around quality: which answers better, which hallucinates less, which is easier to maintain.
That is the wrong place to start. In most of the projects we see, the decision does not break on quality —all three can deliver acceptable quality— but on cost per task. And there the three options behave radically differently depending on your volume. The one that looks cheapest in the demo becomes the most expensive in production, and vice versa.
The three options, in one sentence of cost
Long context. You put the relevant knowledge straight into the prompt on every request. Zero infrastructure, zero preparation. But you pay for those tokens every single time you ask. It is the “copy and paste the whole manual” before every query.
RAG (retrieval-augmented generation). Instead of sending all the knowledge, a retriever pulls only the fragments relevant to each query, and those —and only those— travel in the prompt. The context per request stays small. In exchange, you build and maintain a pipeline: chunk the documents, generate embeddings, a vector store, the retrieval logic.
Fine-tuning. You adjust the model itself with your data. You pay a training cost upfront, but afterwards the knowledge (or the style, or the format) lives inside the model, so the prompt on each request can be much shorter: you no longer need to resend instructions or examples.
Where each one is paid for
The key is to separate setup cost (once) from cost per task (every request, forever). It is the same logic as cost per task: what sinks your budget is not the list price, it is what you multiply by your monthly volume.
Long context phas near-zero setup cost and the highest cost per task of the three. If you drag 8,000 tokens of manual into every query and resolve a hundred thousand queries a month, you have paid for that manual a hundred thousand times. Cheap to set up, expensive to run.
Fine-tuning flips the split: high setup cost (preparing the dataset, training, and retraining every time the knowledge changes), but the lowest cost per task, because the prompt slims down. It is the “pay a lot once to pay little many times” option.
RAG sits in the middle on both dimensions: a moderate setup (the retrieval pipeline) and a low cost per task, because you only send the fragments you need, not everything. It also has a cost the other two don’t show as clearly: each query also pays for retrieval (embedding the question, searching the vector store), though this is usually small next to the savings in model tokens.
An example with numbers
Take a task that needs to lean on a knowledge base. A 300-token answer in all three cases. Illustrative prices of $2 per million input and $12 per million output.
| Approach | Input tokens per task | Cost per task |
|---|---|---|
| Long context (whole manual) | 8,000 | $0.0196 |
| RAG (only relevant fragments) | 1,500 | $0.0066 |
| Fine-tuning (minimal prompt) | 500 | $0.0046 |
At first glance, fine-tuning wins. But that is only half the picture. If you run a thousand tasks a month, the difference between long context and fine-tuning is about $15 a month: nowhere near enough to justify the cost of preparing and maintaining a fine-tuned model. If you run a million, that same difference is around $15,000 a month —$180,000 a year— and suddenly fine-tuning pays for itself in weeks.
The break-even point is not set by which option is “better”, but by how many times you will repeat the task.
How to decide without instrumenting half your team
Three questions sort out almost every case:
What is your volume? At low volumes, long context almost always wins: the cost per task is irrelevant next to the engineering time you save. Don’t build a RAG pipeline to resolve a thousand queries a month.
How often does the knowledge change? If your base changes daily —prices, inventory, policies— fine-tuning is a trap: constant retraining is expensive and slow. RAG shines exactly here, because updating is as simple as changing the indexed documents. Fine-tuning fits when what you teach is stable: a tone, an output format, a domain jargon.
What are you putting in, knowledge or behaviour? RAG is for knowing things (facts, documents, data that gets looked up). Fine-tuning is for behaving a certain way (answering in a style, following a structure, speaking your technical language). Confusing them is the most expensive mistake: trying to load a changing knowledge base via fine-tuning, or forcing a rigid format by stretching the prompt on every request.
And they are not mutually exclusive. The most common combination in production is fine-tuning for behaviour + RAG for knowledge: a model tuned to answer the way you want, fed with fragments retrieved on the fly. You pay the fine-tuning setup once, keep the knowledge cheap with RAG, and the prompt per task stays short on both ends.
An honest caveat
The numbers in this article are an example to illustrate the mechanism, not a measurement of your case. The real break-even point depends on your volume, on how much context each request drags today, and on how often your knowledge changes. It might be at ten thousand tasks a month or at ten million.
What does not change is the order of the decision: first measure how much a task costs you today and how many you do, and only then choose an architecture. Choosing by what’s fashionable —“everyone does RAG”— instead of by your numbers is the most common way to end up overpaying for an architecture your volume doesn’t justify.
This piece is part of the cluster around cost per task of AI, the guide that anchors everything else.
At e-ficient we measure cost per task for companies already running AI in production, including how much each architecture would cost you at your real volume. The initial audit is free if you start a plan with a three-month commitment and returns a diagnosis within 72 hours.