Fine-Tune, RAG, or Prompt: the Data-Governance Tradeoff

There are three ways to give a general model your domain. You can fine-tune it, baking your data into the weights. You can retrieve, fetching your data at runtime and handing it to the model as context. Or you can prompt, describing your domain in the instructions each time you call. Most teams pick between them on capability and cost: which one makes the model most accurate, for the least money and effort.

However, in a regulated enterprise those are the secondary questions. The primary one is where your regulated data ends up living, because that single fact decides whether you can audit the system, delete a person's data on request, and update your domain knowledge without a retraining project. The choice between fine-tuning, RAG, and prompting is a data-governance decision wearing an engineering costume. Pick on accuracy alone and you can back yourself into an architecture where a routine right-to-erasure request means retraining a model.

The way to choose well is to stop asking which technique is most capable and start asking where each one puts your regulated data, and what that location costs you when a regulator, a customer, or your own change process comes calling.

Where the data lives: baked in, retrieved, or in-context

The three approaches differ on one axis that governs everything else: the persistence and location of your regulated data relative to the model.

Fine-tuning bakes the data into the weights. You take examples from your domain, often containing real records, and you train the model on them. After training, that information is diffused across billions of parameters. It is no longer a document you can point at. It is a statistical residue distributed through the model, and the model may reproduce it in an output without ever "retrieving" it, because it was never stored as a retrievable thing. This is the most intimate form of teaching, and the least reversible.

RAG keeps the data in a store the model reads at runtime. Your regulated content stays in a database or index that you own. At query time, the relevant piece is fetched and placed in the context window, used for that one response, and then gone from the model's perspective. The model's weights never change. The data lives where you put it, under your access controls, and the model borrows it briefly.

Prompting puts the data in the instruction, in-context, per call. Nothing persists. The domain knowledge is described or supplied in the prompt, used for that single inference, and discarded. It is the most transient option: the model learns your domain for the length of one API call and forgets it completely afterward.

Same goal, three radically different data-governance profiles. The rest of the decision follows from this one distinction.

Auditability and right-to-erasure: the questions a regulator will actually ask

Two obligations expose the difference sharply, and both are routine in a regulated shop.

Consider auditability. A regulator asks: what information did the system use to reach this decision, and can you show me? With RAG, the answer is clean. You retrieved these specific documents, at this index version, and the decision was derived from them; the evidence trail names the sources. With prompting, it is similarly clean: the context was assembled from these inputs, here they are. With a fine-tuned model, the honest answer is uncomfortable. The decision drew on knowledge diffused through the weights during training, and you cannot point at the specific record that shaped it. You can describe the training set; you cannot trace a specific output back to a specific input. For a process where every decision needs an inspectable basis, that is a structural problem, not a documentation gap.

Now consider right-to-erasure. A person exercises their right to have their data deleted. If their record sat in a RAG store, you delete the record and it is gone from future retrievals, immediately and provably. If it was only ever in prompts, it was never persisted; there is nothing to erase. But if their data was baked into a fine-tuned model, deleting it is genuinely hard. The information is entangled in the weights, and there is no clean operation to remove one person's contribution short of retraining without them, or applying machine-unlearning techniques that are still maturing and hard to prove to a supervisor. A right you must honor in days can collide with a model you retrain in weeks.

Neither of these makes fine-tuning wrong. They make it a commitment whose full cost only shows up later, at exactly the moment a regulated organization can least afford a surprise.

Update cost: how fast can your domain knowledge change?

There is a third practical axis, and it is the one that bites operationally rather than legally: how expensive is it to change what the model knows?

Domain knowledge in a regulated business is not static. A policy is revised, a threshold moves, a rule is superseded. How each approach absorbs that change differs by orders of magnitude. Updating a RAG corpus is a document operation: replace the source, re-index, done, and the change is live on the next query. Updating a prompt is a text edit and a redeploy of the instruction. Updating knowledge that lives in fine-tuned weights means a new training run, revalidation, and a controlled release of a new model version. The first two are hours. The third is a project.

This is where staleness becomes a governance risk rather than an inconvenience. If your regulated knowledge changes faster than you can retrain, a fine-tuned model that encodes that knowledge is drifting out of compliance between training runs, and nothing in its behavior announces it. RAG and prompting keep the fast-changing, regulated content in a layer you can update at the speed the rules actually change.

A decision framing for a regulated team

None of this reduces to "always use RAG." It reduces to matching the technique to what the data is and how it behaves.

The framing that holds up in front of a risk committee is to separate two things the model needs. It needs skills, the stable, general competence to read a document, follow a format, reason through a structured task. And it needs facts, the specific, changeable, often regulated content it reasons over. Put the two in different places.

Fine-tuning is defensible for teaching durable, non-sensitive skills: a house style, an output schema, a reasoning pattern that does not change and carries no personal data. It is a poor place to put facts, and a dangerous place to put regulated facts, precisely because of auditability, erasure, and update cost. Regulated facts belong in RAG, where they stay inspectable, deletable, and current, under access control enforced at retrieval time. And prompting carries the per-call, transient context that neither persists nor needs to. Most robust regulated systems use all three deliberately, with a clear rule about which kind of information is allowed to live where, rather than reaching for whichever one looked most accurate in a demo.

So the question to settle before you pick a technique is not "which approach makes the model most accurate?" It is what regulated data would this approach cause us to store, where would it live, and can we audit it, delete it, and update it at the speed our obligations demand? Answer that first, and accuracy becomes a choice you make inside a safe design instead of a reason you ended up in an unsafe one.

If you want to map which of your domain knowledge belongs in weights, in retrieval, or in the prompt, book a meeting and bring us the process with the most regulated data in it.