RAG, Fine-Tuning or Agents: Picking the Right Layer and the Order to Build It

Two questions come up in almost every LLM project sooner or later. One is whether to retrieve documents or fine-tune the model. The other is whether the product needs an agent or just a smarter prompt. Both get answered badly surprisingly often, and not only by beginners. Experienced engineers have spent weeks tuning a model to memorize facts that a retrieval call would have supplied in an afternoon. Other teams have wrapped an agent framework around what was really one question-answering endpoint, simply because agents were the topic in every standup that quarter.

The less exciting reality is that retrieval, fine-tuning and agents are not rivals for the same job. Each answers a separate problem, and systems that run in front of real users commonly combine two of them or all three, with every piece doing what it is good at. What follows covers how each works underneath, what it costs, how teams misuse it, how the pieces fit together and a small project that shows the order to build them in.

What each one actually does

Retrieval-augmented generation

Retrieval leaves the model untouched. It only changes what the model gets to read before answering.

When a question arrives, the system turns it into an embedding, a numeric representation of its meaning, and looks in a vector database such as Pinecone, Qdrant, FAISS or Azure AI Search for the stored chunks that are closest in meaning. Those chunks are placed in the prompt as context, and the model answers from that material instead of from whatever it absorbed during pretraining.

The value is in the failure it prevents. Models invent details most readily when forced to answer from internal memory, particularly on topics that are recent, obscure or specific to one company. Giving the model reference text on every call means it reasons over current material and does not guess.

The pipeline has a fixed shape: ingest documents, split them into chunks, embed the chunks, store the vectors, run a similarity search (often combined with keyword search), optionally re-rank the candidates, assemble the prompt and generate.

Fine-tuning

Fine-tuning differs in kind, not only in degree. A pretrained model keeps training on a curated set of your own examples, usually pairs of input and desired output that demonstrate the exact behavior wanted. The weights change, or with LoRA and QLoRA a small set of adapter weights on top of a frozen base changes.

It works well for teaching manner: a tone of voice, a rigid output format, a particular way of working through one type of problem, or the handling of edge cases in a field where thousands of labeled examples exist. The aim is to reshape default behavior so the same instructions need not be repeated in every prompt, rather than to add facts.

It works poorly for anything that keeps changing. A product catalog that updates weekly cannot sensibly live in the weights, because knowledge freezes when training ends and each update means another run.

Agents

An agent places the model inside a loop: observe, reason, act, observe the result, reason again, and continue until a goal is reached or a stopping rule fires. The model no longer produces a single answer. It decides what to do next, frequently by calling tools such as APIs, functions, databases and other services, and feeds their output into its following decision.

That loop is what turns a system that answers questions into one that completes tasks. LangGraph, CrewAI and a hand-written ReAct loop all implement a version of it: the model writes a thought, selects a tool, the tool runs, the result returns to the context, and the cycle repeats until the agent judges itself finished or reaches a step limit. A step limit should always be set.

A compact way to keep the three apart:

• Retrieval supplies the right information.
• Fine-tuning supplies the right behavior.
• Agents supply the right degree of autonomy.

Side by side

The differences show up across several dimensions:

• Knowledge updates. Retrieval is live, since refreshing the vector store is enough. Fine-tuned knowledge is frozen until the next training run. For agents it depends on whatever they retrieve or call.
• Cost. Retrieval is moderate: embeddings, storage and extra input tokens. Fine-tuning is high because of data curation, compute, GPU time and evaluation. Agents range from moderate to high because every task involves several model calls plus tool charges.
• Complexity. Retrieval needs chunking, embedding and tuning of the search. Fine-tuning needs data preparation, training infrastructure, evaluation loops and protection against overfitting. Agents need planning, tool integration, error handling and loop control.
• Time to set up. Days for retrieval, weeks for fine-tuning, and days to weeks for agents depending on the workflow.
• Best suited to. Question answering, document search and grounding for retrieval. Tone, formatting and domain reasoning style for fine-tuning. Multi-step jobs, automation and tool use for agents.
• Latency. Retrieval adds one hop, typically 50 to 300 milliseconds. Fine-tuning adds nothing at inference compared with the base model. Agents often take several hops and can need seconds per task.
• Maintenance. Retrieval means keeping the knowledge base current. Fine-tuning means retraining periodically as needs drift. Agents mean updating workflows, tools and prompts.
• Typical failure. Bad retrieval leads to irrelevant context and a bad answer. Overfitting makes a tuned model brittle on unfamiliar input. Agents loop, call the wrong tool or run up costs without limit.

Three situations, one for each

A support chatbot over product docs fits retrieval. A software company with 400 pages of documentation, changelogs and policies could fine-tune on that text, but the tuned model would be stale by the next release. Embedding every chunk into a vector store avoids that. When a customer asks whether the API supports batch requests, the system pulls the three most relevant sections, the model answers from them and cites the page. After the next sprint changes the docs, re-embedding refreshes the answers with no retraining.

A house-style writing assistant fits fine-tuning. A media company wants summaries in an exact style: a particular sentence rhythm, no first person and a set number of takeaways at the top. Prompting gets close but drifts over long sessions. Training on about 2,000 human-edited summary pairs makes the model produce on-brand text reliably, with no need to paste the style guide into each request.

Candidate screening fits an agent. A recruiting startup wants to say screen this candidate and book a call if they qualify and have the rest happen. The agent reads the resume, checks it against role requirements through an internal scoring API, looks up the hiring manager's calendar, writes and sends the scheduling email and records the outcome in the applicant tracking system. One instruction covers all of it, and nobody touches the individual steps.

The same split appears repeatedly in practice. Retrieval-heavy designs handle grounding, agents handle orchestration, and fine-tuning is used only when prompting cannot hold behavior steady.

How they fit together

The three are not exclusive layers. The most capable systems chain them, with a request flowing from the agent that plans, through retrieval that supplies facts, to a tuned model that phrases the output.

User Query
    ↓
   Agent (decides what needs to happen — plan the steps)
    ↓
   RAG (retrieves relevant knowledge, if the step needs facts)
    ↓
   LLM (fine-tuned, if tone/format/behavior consistency matters)
    ↓
   Action (respond to user, call a tool, trigger a downstream workflow)
    ↓
   Agent (evaluates result → loop again or stop)

In a support setting this plays out as follows. A message arrives and the agent classifies the intent, concluding that product information is needed. Retrieval pulls the relevant documentation chunks. A fine-tuned model drafts the reply in the brand voice. The agent then judges whether confidence is high enough to send automatically or whether a human should take over, and finally logs the exchange in the CRM.

Remove any piece and a gap appears. Retrieval without an agent cannot decide when to look something up and when to act. An agent without retrieval reasons blind about anything beyond its training data. A fine-tuned model without retrieval produces confident, nicely formatted answers that may be out of date.

What each costs

The numbers below describe direction only, since real figures vary with scale and provider.

For retrieval, the upfront cost is embedding the corpus, either once or incrementally as the document volume grows. Running costs are vector database hosting on a pricing tier plus extra tokens for the retrieved context on each request. It is the cheapest lever for the value delivered, which is why most teams should begin there.

For fine-tuning, the heavy upfront item is often data: collecting and labeling good training examples is expensive. Compute for training and evaluation cycles come next. Afterwards there is retraining whenever behavior must shift, and hosting for a custom model unless LoRA on a shared base is used. It demands the largest initial investment and tolerates mistakes least, because poor training data writes poor behavior directly into the weights.

For agents, setup involves workflow design, tool integration and testing of failure paths. The running cost is the sneaky part, since each agent step is another model call. A five-step task can cost five to ten times as much as a single retrieval query, and a loop without a cap on iterations can run away entirely. Cost tracks the complexity of the task and not just the volume of usage.

The practical reading is that retrieval is usually the cheapest route to the biggest drop in hallucination. Fine-tuning pays for itself only when consistent behavior is a true product requirement and not a nice extra. Agents justify the per-task price only when the job needs several decisions and actions, and not when one well-grounded call would do.

Rules of thumb for choosing

• Pick retrieval when the data changes often. Docs, policies, catalogs, pricing and inventory that move weekly or daily suit it well and suit fine-tuning poorly.
• Pick fine-tuning when behavior matters more than knowledge. If the model knows the domain but keeps using the wrong tone, format or reasoning pattern despite good prompting, the problem is behavioral.
• Pick an agent when the task needs several steps or actions in the real world. Answering a question does not call for one. Working out what must happen and then doing it across systems probably does.
• Combine retrieval and fine-tuning when grounded facts and a steady voice are both required, which is very common in customer-facing products.
• Combine retrieval and an agent when the task needs facts plus decisions, with the agent choosing what to look up and when to act on it.
• Skip the agent when it is only fashionable. If a well-built prompt and one retrieval call settle the matter, an agent adds latency, cost and room for failure with no gain.

The same logic fits in two short flowcharts. The first checks whether the answer depends on information that changes often.

Does the answer depend on information that changes often?
   YES → RAG
   NO  ↓

The second asks about behavior consistency and about whether the task spans multiple steps or real actions.

Does the model need to consistently behave/sound a certain way,
and prompting alone isn't holding that consistency?
   YES → Fine-tuning
   NO  ↓Does the task require multiple steps, tool calls, or real actions
(not just answering a question)?
   YES → Agent (likely combined with RAG, and fine-tuning if voice matters)
   NO  → A single well-prompted LLM call is probably enough

A small project showing the order

Take an internal HR assistant.

1. Start with retrieval. Embed the employee handbook, leave policy and benefits documents into a vector store. Employees can then ask how many sick days they get and receive a grounded, cited answer. This handles roughly 70 percent of the use case and takes a few days.
2. Add fine-tuning only if a gap appears. Suppose HR wants every reply to carry a specific empathetic tone and a strict format of here is the policy, here is what it means for you, and prompting keeps drifting from that. A small model tuned on a few hundred curated question and answer pairs in that style closes the gap.
3. Add an agent for the multi-step cases. A request to take next Friday off is not a lookup. It means checking the leave balance, checking the team calendar for conflicts, submitting the request and confirming back to the employee, each as a tool call. The agent uses the retrieval layer for policy questions along the way and the tuned model for the tone of the final message.

The order is deliberate. Retrieval comes first because it is cheapest and covers the most ground. Fine-tuning follows only once a real behavioral gap shows. Agents come last, and only when the task truly needs multi-step action.

Frequent mistakes

• Fine-tuning where retrieval was needed. Teams try to bake knowledge into the weights when the actual problem is missing access to current information. The result is slower, far more expensive and out of date as soon as it ships.
• Agents with no defined workflow. A vague goal with unrestricted tools makes an agent wander, loop and behave unpredictably. Narrow tools, clear stop conditions and a firm ceiling on reasoning steps keep it in line, whereas telling it to work things out is no substitute for a defined process.
• Ignoring latency and cost. Each retrieval hop and each agent step brings another round trip and more tokens to pay for. A demo can look excellent and then turn out in production to need 12 seconds and six model calls for one answer.
• Reading the options as a ladder. None of the three is more advanced than the others. The best system is the simplest one that does the job, which is often retrieval alone or retrieval plus a lightly tuned model, well before an agent stack is warranted.
• Skipping evaluation. All three degrade quietly: weak retrieval chunks, overfit tunes, or agents that take a wrong action without anyone noticing. Evaluation sets should exist before any of them scales.

Direction of travel

The industry is clearly heading toward agentic workflows, meaning systems that plan, act and follow up without being prompted each time. The step after that is the hybrid: retrieval to ground answers, tuning to stabilize behavior, and long-lived memory so an agent can recall earlier exchanges instead of starting blank each session.

There is a visible shift from chatbots to what some now call AI employees, systems that carry out ongoing work across tools and over time instead of waiting for a question. Reaching that point has little to do with choosing the flashiest technique. It depends on understanding what each part does well and assembling them deliberately, beginning with the simplest layer that solves the problem.

The choice was never retrieval versus fine-tuning versus agents. The real question is which combination a given system needs, and in what order to build it.