A company tells us: “We want to use AI on our own data.” Sounds simple, right? Then the real question lands, and it’s almost always the same one. Should we use RAG, fine-tune a model, or build an agent?
Here’s the thing: these three approaches solve different problems, cost wildly different amounts, and fail in completely different ways. Pick wrong and you’ve just signed up for months of wasted effort. (We’ve watched it happen. It’s painful.) So let’s walk through the framework we actually use, with examples from projects we’ve seen succeed and, let’s be honest, fail.
Key takeaways
- RAG injects your knowledge at query time, and it stays current by design.
- Fine-tuning shapes behavior and format. Do not use it to store facts.
- Agents are for genuinely open-ended, multi-step work, and they fail by compounding small errors.
- Exhaust the simple option before buying the complex one.
- Real production systems usually combine all three, deliberately.
The one-paragraph version of each approach
RAG (retrieval augmented generation) keeps the model frozen and injects relevant documents into the prompt at question time. Ask about your refund policy, the system fetches the policy text, and the model answers from that. Fine-tuning continues training the model on your examples, so its behavior, style or domain skill changes for good. Agents wrap the model in a loop with tools, search, databases, code execution, APIs, so it can plan, act and check its own work over multiple steps. Three tools, three jobs.
When RAG is the right answer
RAG wins when your problem is knowledge, not behavior. Your model already writes beautifully. It simply doesn’t know your internal documentation, your product catalog or last quarter’s reports. RAG fixes that without touching the model itself.
The classic winning use cases? Support chatbots grounded in a help center. Internal search that actually answers instead of listing links. Research assistants that work over a document corpus. And the advantages are decisive for most teams: no training cost, instant updates when documents change, and citations that let users verify every answer.
Where does RAG fail? In retrieval, almost always. If the right paragraph never gets fetched, the model either guesses or refuses. So before you blame the model, invest in chunking, good embeddings and hybrid search. In our experience, eighty percent of disappointing RAG systems have a retrieval problem, not a generation problem. Read that again before you switch models.
When fine-tuning is worth the trouble
Fine-tuning changes how the model behaves, not what it knows. Reach for it when you need a consistency that prompting can’t hold steady: classifying support tickets into your exact taxonomy, rewriting medical notes in your house style, generating SQL in your company’s dialect, or compressing a long system prompt into the weights to cut latency and cost.
Now, the mistake that burns teams. Don’t fine-tune to teach facts. Facts baked into weights go stale, can’t be cited, and come back unreliably. We’ve seen the story play out: a team fine-tunes on their documentation, ships it, and three months later the model confidently recites the old refund policy to a customer. That knowledge belongs in a retrieval layer.
The good news? Costs have collapsed. Fine-tuning a capable open model for a narrow task can run under a hundred dollars with parameter-efficient methods, and hosted options from OpenAI, Mistral and others remove the infrastructure headache entirely. The real investment is the dataset: a few hundred to a few thousand high-quality examples, carefully reviewed by a human.
When you actually need an agent
An agent earns its complexity when the task requires decisions and actions in sequence. Research this prospect and draft the outreach. Reconcile these invoices against the bank feed. Monitor this competitor and brief me weekly. Sound like your workflow? Then maybe.
But ask yourself one question first: can the steps be fully specified in advance? If yes, build a boring, reliable pipeline. Agents add value exactly when the steps can’t be known ahead of time.
And here’s why agent projects fail predictably: error compounding. If each step is right ninety percent of the time, a ten-step task succeeds barely a third of the time. Ouch. Guardrails, verification steps, human checkpoints on irreversible actions, and ruthless scope limits are what separate the demos from the deployments. Start with read-only agents that recommend. Graduate to agents that act.
The decision framework we actually use
- Can prompting with good examples solve it? Try this first. Always. It costs an afternoon.
- Is the gap knowledge or behavior? Knowledge gaps call for RAG. Behavior, format or style gaps call for fine-tuning.
- Does the task require multi-step action in the world? That’s the agent case, and only that case justifies the orchestration overhead.
- Combine deliberately. Most production systems mix all three: a fine-tuned model, grounded by RAG, inside an agent loop with a few sharp tools.
What this all costs in practice
Let’s talk real numbers. A solid RAG prototype on your documents: days of work, modest API bills, a vector database you can start for free. A fine-tuning run: a dataset project first, then trivial compute. An agent that survives contact with production? Budget weeks for evaluation, guardrails and monitoring, because the building is the easy part.
Anyone selling you the opposite ordering has a demo, not a system.
How we developed this framework. This guide distills patterns from real deployments we have built or audited, from two-person startups to enterprise teams, plus public post-mortems from engineering blogs we trust. We revisit the recommendations as tooling evolves. Details on our methodology page.
The evaluation habit that separates pilots from products
Whichever approach you choose, the teams that ship share one habit the others lack: they measure. Before building anything, they assemble a test set of real questions with known-good answers. Fifty to two hundred of them. After every change, a prompt tweak, a new embedding model, a fine-tune, they re-run the set and compare.
Unglamorous? Absolutely. But it’s the entire difference between a demo that impressed the board and a system that survives real users.
A few specifics worth stealing. For RAG, measure retrieval separately from generation: was the right passage fetched, and was it used correctly? For fine-tunes, hold out examples the model never saw, and check that narrow training hasn’t eroded general skill. For agents, track end-to-end task completion, not step counts. A spreadsheet and discipline are enough to start. What matters is the honesty to look at failure rates before your users find them for you.
The bottom line
Let’s bring it home. RAG for knowledge. Fine-tuning for behavior. Agents for multi-step action. And prompting before all of it. The boring answer is usually the right one, and the boring answer ships.
If you’re still at the stage of evaluating assistants that already bundle these capabilities, our assistant comparison shows how the mainstream products implement these very ideas under the hood. Start there, and when your needs outgrow them, you’ll know exactly which approach to reach for.