RAG, fine-tuning or an agent: what does a company need?

Written by the editorial team · 12 November 2026 · Updated12 November 2026

RAG hands a model the right documents at the moment it answers. Fine-tuning adjusts how the model behaves (format, tone, consistency). An agent picks its own tools to carry out a task in several steps. To answer from a company's documents, start with search over those documents; add an agent when the task chains several actions, and fine-tuning last, if the model's behaviour stays inconsistent.

One vendor talks about RAG, another about fine-tuning, a third about agents. The three words show up in the same slide decks, yet they name three different levers. This article defines each from Anthropic’s and OpenAI’s own documentation, which we read, and then says which to look at first.

What is RAG?

RAG stands for retrieval-augmented generation. OpenAI sums it up as retrieving content to augment the model’s prompt before generating an answer. When the question comes in, the model is handed the passages of your documents that matter for the case.

In OpenAI’s guide, RAG deals with a context problem: the model does not know your documents, or knows an outdated version of them. The trap lies in the retrieval. The guide warns that too much irrelevant context drowns out the useful information and causes hallucinations. A RAG is only as good as its search: finding the right contract, the right version, the right folder.

What is fine-tuning?

Fine-tuning continues training a model on a smaller dataset specific to one task. According to OpenAI, it targets behaviour: consistency of answers, format, tone, reasoning. It can also reach the same accuracy with fewer tokens or a smaller model.

The guide recommends exhausting prompt work before moving on to RAG or fine-tuning. It flags a common trap: training examples that differ subtly in format or form from what the model will receive in production. The model then learns the wrong habits.

What is an agent?

Anthropic separates two things. A workflow orchestrates models and tools through predefined code paths. In an agent, “LLMs dynamically direct their own processes and tool usage”: the model decides what to do next.

On the technical side, Anthropic’s tool use documentation describes the mechanism. The model returns a structured call, the application runs it and hands the result back. Searching a document, reading a table, writing a file: each action is a tool.

Anthropic adds a limit that many sales decks leave out. Agentic systems trade latency and cost for better task performance, and their autonomy opens the door to errors that compound from one step to the next.

Which one fits your need?

These levers stack. According to OpenAI, RAG and fine-tuning can be combined, and Anthropic describes an agent as a model with retrieval, tools and memory. Document search then becomes one of the agent’s tools.

Order matters. Anthropic advises finding the simplest solution and adding complexity only when needed, which, it writes, “might mean not building agentic systems at all”. Agents fit when you need flexibility and model-driven decisions at scale.

How do you recognise your own case?

Three typical situations help you decide.

An assistant gets things wrong because it does not know a document, a price grid or an internal procedure: that is a context problem, so a search problem. An assistant knows the information but words it differently each time, in a format your team has to rework: that is a behaviour problem, and a more precise prompt is the first remedy. A request needs three documents opened, two versions compared, then a file produced: neither search alone nor training chains that together, and that is an agent’s job.

Anthropic’s tool use documentation adds a useful detail: by default, the model decides on each turn whether to call a tool or answer directly. It answers directly for stable knowledge or conversation, and calls a tool when the answer is not already in context. An agent is therefore not a model that “does everything by itself”. It is a model given well-described tools and a framework for using them.

Which traps should you avoid?

• Launching a fine-tuning so the model “knows” your documents. OpenAI’s guide puts knowledge on the RAG side.
• Plugging in a RAG without ever measuring what the search returns. Bad passages produce bad answers, delivered with confidence.
• Building an agent for a task that fits in a single call.
• Letting the agent send its output without review. Anthropic recommends extensive testing in sandboxed environments and guardrails.
• Training on examples that look nothing like the real requests.

And with us?

We publish this blog and we are a forward deployed agency: three Télécom Paris engineers specialised in AI, who build custom agents and also take on machine learning and data missions.

Our agents read your documents on SharePoint, OneDrive or Dropbox, read-only, in the folders you choose, and keep no file. They cite their sources and write Word and PowerPoint documents. Nothing goes out without a person at your company approving it. The first call lasts 30 minutes, is free and carries no commitment; two weeks later, a first agent is running. For more, see the page on AI agents for companies and the one on the context layer.