Diagnosing LLM Output Problems: When to Prompt, Retrieve or Fine-Tune

The first version of any AI feature produces output that is not quite right. You have three levers to fix it: change the instructions, give the model the documents it is missing, or retrain it on your own examples. They differ enormously in cost, from minutes of work to weeks of data collection, and each one fixes a different kind of failure. This guide gives you a symptom-first way to pick the right lever, so you stop spending weeks on a fix the problem never needed.

Treat the model like a capable new hire

A useful mental model is to think of the model as a very sharp employee on their first day. They know a remarkable amount about the world in general and nothing at all about your company: not your products, not your policies, not the way your team likes answers written. They will make mistakes, just as any talented newcomer does.

You can help that new hire in exactly three ways. You can brief them better, you can hand them the reference material, or you can send them on a training course. Those map to prompting, retrieval and fine-tuning, and they are ordered from cheapest to most expensive. The skill lies in matching the intervention to the problem, so it makes sense to look at them in that order.

Prompting: rewrite the briefing first

Always start here. A prompt is simply the instructions you send with the request: the task itself, one or two examples of a good answer, who the answer is for and the format you expect back. Changing it costs nothing and takes minutes. A large share of complaints that "the AI is bad" turn out to be complaints that the prompt was vague.

Suppose your feature's replies come back long-winded and stiff. Nothing needs retraining. You add an instruction such as "reply in three sentences, in a warm, plain tone", paste in one good sample reply, and the problem is usually solved in a single iteration. The standing instructions you set once for every request are called the system prompt; the sample answers you include to demonstrate a pattern are called few-shot examples.

Prompting has a hard boundary, though: instructions cannot supply knowledge the model has never seen. If the new hire has never been shown this quarter's figures, telling them to "be more precise" will not produce those figures. When the real gap is information, you need the second lever.

A quick check before moving on

Before you conclude that prompting has failed, make sure you have tried the obvious improvements: state the output format explicitly, give at least one concrete example, say what to do when the answer is unknown, and test against a small fixed set of real inputs rather than one or two hand-picked cases. Without that fixed set, it is hard to tell whether a change actually helped.

Retrieval: hand over the files

Retrieval-augmented generation, usually shortened to RAG, gives the model access to your documents at the moment it answers: the current price list, the returns policy, a particular customer's order history. It addresses any knowledge that is missing, changes often or is private to your organisation. When a document is updated, the answers change with it, and no retraining is involved. It is the right choice whenever the knowledge belongs to you, moves frequently or needs to be cited. The distinction between what a model has memorised in its weights and what it looks up at answer time is covered in more depth in how AI memory, context, embeddings and model weights differ.

In the new-hire picture, you are no longer lecturing them; you are giving them the handbook and letting them check it before they respond. They can now answer questions about things that changed long after the model was trained, and they can point to exactly where each answer came from.

The price is higher than a prompt tweak. You are building a small pipeline that stores documents and searches them by meaning, which typically means days of work rather than minutes. That is still far cheaper than fine-tuning, and the system stays current as your documents do. It also adds moving parts you now have to maintain: how documents are split, how search quality is measured and what happens when nothing relevant is found.

Retrieval has its own clear boundary. A document fills gaps in what the model knows; it does not change how the model behaves. Giving someone the manual does not alter their writing style or their judgement. For that, you train them.

Fine-tuning: send it on a training course

Fine-tuning retrains the model on many examples until a style or skill becomes automatic. It is the lever for consistent behaviour that prompting keeps failing to deliver, or for the moment when your prompt has grown into a page of rules and is still unreliable. Imagine needing every reply to follow a very specific brand voice across a million conversations without a single one drifting. Training on thousands of examples can make that behaviour the default, with no instructions required.

The point people most often get backwards is this: fine-tuning changes how the model behaves, not what it knows. It is slow and expensive, it demands a substantial set of example data, and anything factual you bake in goes stale the moment those facts change. So you should not use it to store facts; that is what retrieval is for. It is the training course among the three options: powerful when the problem is genuinely behavioural, and wasted on anything a clearer briefing would have fixed. For a worked look at the economics of this decision, see costing a fine-tuned model against an API call.

Choosing by symptom

Start with a single question: what exactly is wrong with the output?

• The format or tone is off, or the model ignores parts of your instructions. That is a briefing problem, so improve the prompt.
• A fact is missing or outdated, the model cites the wrong version of something, or you need it to show its source. That is a knowledge problem, so add retrieval.
• The facts are right, but a behaviour will not hold however you phrase the request, and you need it to be dependable across thousands of responses. That is a behaviour problem, so consider fine-tuning.

The symptom chooses the fix. Two further points are easy to overlook.

The levers stack rather than compete

Nearly every feature begins with a prompt. Many later add retrieval when they need live or private facts, and a smaller number add fine-tuning for a behaviour the prompt cannot secure. You are usually deciding which layer to add next, not choosing one approach for all time. A fine-tuned model still receives a prompt, and a RAG system still depends on instructions telling the model how to use what it retrieved.

Go only as far as the symptom requires

Because the options run from cheap to expensive in that order, stop at the first one that solves the problem. The same reasoning works before anything is built: if the job depends on facts that change, plan for retrieval; if it needs one precise behaviour at very high volume, fine-tuning may eventually be justified; everything else starts with a prompt.

The costly mistake: fine-tuning to teach facts

The error worth calling out explicitly is reaching for fine-tuning to make the model know something. It sounds like the serious, engineering-heavy choice, which is exactly why teams pick it first. But a fact that keeps changing does not belong in the model's habits; it belongs in a document the model can look up. Get this backwards and you can spend weeks teaching the model a fact that is already wrong again by the time you launch, with no easy way to show where an answer came from.

A related trap is fine-tuning to fix what is really a vague prompt. If you have not yet tried explicit formatting instructions and a few good examples against a fixed test set, you do not yet know whether you have a behaviour problem at all.

Key takeaways

• Diagnose before you spend: identify whether the failure is about instructions, knowledge or behaviour.
• Instructions, tone and format problems are solved with prompting, in minutes.
• Missing, changing, private or citable facts call for retrieval, which takes days to set up and stays current.
• Only a stubborn behaviour that no prompt can make reliable justifies the weeks and data that fine-tuning demands.
• Treat the three as layers you add in order of cost, and never use fine-tuning as a place to store facts.