Fine-tuning versus prompting: when each one is worth it

Written By: BrainSoft In AI & Research

Almost every LLM feature starts the same way: someone writes a prompt, it works on three examples, and then it falls apart on the fourth. The reflex is to reach for fine-tuning. Sometimes that is right. Often it is a way to spend two weeks and a few hundred dollars to fix a problem that a better prompt and a validation loop would have caught in an afternoon.

Short answer: prompt first, always, and treat fine-tuning as a fix for a specific failure you can name. Prompt when the task is described by instructions and examples fit in the context window. Fine-tune when you need a consistent output shape, a tone or domain behaviour the base model will not hold, latency and token cost that prompting cannot reach, or when you have thousands of labelled examples and no room to send them at inference time.

What prompting actually gets you

A prompt is a program you can edit in seconds. That matters more than it sounds. Requirements change in week two, the edge cases only show up in production, and the ability to redeploy a behaviour change without a training run is worth a lot. Few-shot examples, a strict output contract, and a couple of validation retries cover most classification, extraction and summarisation work.

The trick is to stop guessing. Build a small eval set of real inputs with expected outputs, then measure every prompt change against it. Twenty to fifty examples is enough to catch regressions. Without that, prompt iteration is just vibes, and you will not know whether fine-tuning helped either.

def score(prompt, cases):
    hits = 0
    for text, expected in cases:
        out = call_model(prompt, text)
        hits += out.strip() == expected
    return hits / len(cases)

Costs stay predictable too. You pay per token, you can cache the static prefix, and you can swap models when a cheaper one gets good enough. No artefact to retrain, no version of the weights to track.

When fine-tuning earns its keep

Fine-tuning changes the weights, so it changes behaviour that instructions cannot reliably reach. The cases where we have seen it pay off:

• Output format is rigid and the prompt keeps drifting. A model tuned on a few thousand well-formed examples holds the shape far more consistently than a paragraph of rules.
• Domain vocabulary the base model handles badly: internal codes, product names, a legal or medical register it has never seen in context.
• Latency or cost pressure. A short prompt on a small tuned model can beat a long prompt on a large one, which matters at high request volume.
• Long, stable instructions. If your system prompt is three thousand tokens and never changes, baking it into weights removes that cost from every call.
• Privacy or offline constraints. You cannot ship your data to a hosted API, so you train and serve your own model.

Note what is not on that list: teaching the model new facts. Fine-tuning is poor at that. Retrieval is the tool for facts, and it stays current without a retraining cycle. If your problem is “the model does not know our pricing”, that is a retrieval problem, not a training problem.

The costs nobody mentions in the demo

Fine-tuning is not a one-off. You need a labelled dataset, which usually means someone spends days cleaning and reviewing it. You need a held-out split so you can tell improvement from overfitting. You need a serving path: a hosted fine-tune endpoint, or your own GPU box with the operational work that implies. And you need to redo the run when the base model is deprecated, which happens on a schedule you do not control.

There is also the evaluation problem. Once you have a tuned model, comparing it to the prompt version is only meaningful if both were scored on the same eval set. Teams skip this and end up shipping a model that feels better while being measurably worse on the cases that matter. If you want help setting up that loop, our services cover the evaluation and serving side.

A middle option we use a lot: keep the prompt, add retrieval for facts, and only fine-tune once the eval set shows a ceiling that instructions cannot break. That ordering keeps the cheap lever cheap for as long as possible.

A decision rule that survives contact with reality

Write down the failure you are trying to fix. If it is “the model does not know X”, add retrieval. If it is “the model does not follow the format”, tighten the prompt, add a schema, and validate the output. If it is “the model follows the format 90% of the time and we need 99%”, and you have the examples, fine-tune. If it is “this is too slow or too expensive”, try a smaller model with a shorter prompt first, then fine-tune that smaller model if it is close.

The order is not dogma, but it is cheap to follow and expensive to skip. Every hour spent on a clean eval set makes both paths faster, because you stop arguing about which one is better and start measuring it.

Frequently asked questions

#AI & Research