Fine-tuning an AI model

Fine-tuning continues training a pre-trained foundation model on a curated dataset so the model’s weights update. The result is a model that behaves differently: its output format, tone, schema adherence, or narrow-task accuracy shifts to match the training signal. Fine-tuning does not reliably add new factual knowledge. That is the job of RAG. Conflating the two is the single most common failure point in AI PM interviews, and interviewers at AI-native companies (Anthropic, OpenAI, Cohere, Glean) will flag it immediately.

What fine-tuning changes (and what it does not)

Fine-tuning modifies model weights. Every subsequent inference call uses those new weights. That makes fine-tuning a permanent behavioral change, not a query-time augmentation. RAG leaves weights unchanged and injects knowledge at inference time. These are not substitutes: they solve different problems and can be combined.

Fine-tuning is the right tool when you need the model to:

• Match a strict output schema reliably (JSON, structured fields, a specific response template)
• Adopt a brand voice or tone consistently, without a multi-paragraph system prompt on every call
• Perform a narrow task with higher precision than prompt engineering achieves on evals
• Run faster and cheaper at scale by using a smaller model trained to perform like a larger one

Fine-tuning is the wrong tool when you need the model to:

• Know things it was not trained on (use RAG or tool-calling)
• Stay current with fast-changing facts (fine-tuned models freeze the training set; RAG can be updated continuously)
• Experiment rapidly (fine-tune iterations take hours to days; prompt changes are instant)

In 2026, context windows support 200K to 1M tokens. Many tasks that once required fine-tuning can now be handled with long-context prompting and few-shot examples. Always check the prompt-engineering ceiling before committing to a training run.

The 2026 unit economics

In 2026, feasibility is not the question. You can fine-tune a capable model for under $100 in an afternoon. The PM’s job is to answer whether it creates a durable product advantage or just buys maintenance debt.

Concrete numbers for the calculation:

• GPT-4o fine-tuning training costs $25 per million tokens. GPT-4.1 nano is $1.50 per million tokens: over 16x cheaper for simpler tasks where it performs well enough.
• Training cost is multiplied by the number of epochs. A 100K-token dataset trained for 3 epochs costs 3x the per-token rate, roughly $7.50 on GPT-4o.
• Fine-tuned GPT-4o inference is priced at approximately 1.5x the base rate. You pay the premium on every call indefinitely, not just at training time.
• If your fine-tune eliminates a 400-token system prompt, you save roughly $0.12 per 1,000 requests. A training run that costs $0.90 recoups in under a day at 10K requests per day.
• Customer support is the benchmark use case: 2,000 to 10,000 historical ticket-response pairs can yield 35 to 50% better resolution accuracy versus prompt engineering alone, with 40 to 60% lower per-request cost. But ROI timeline is 4 to 8 months at 50,000 or more tickets per month. Below that volume, long-context prompting is almost always the better call.

New foundation models release every 4 to 6 months in 2026. A fine-tuned model can fall behind the capability frontier within one release cycle. Re-tuning on each new base model, re-running evals, and managing rollout is a real PM cost. Budget for it before you commit.

The distillation pattern (2026)

The decision calculus shifted when distillation became a standard pattern. Use a frontier model as the teacher: generate synthetic labeled examples at scale, then fine-tune a smaller model (Llama 4 8B or a GPT-4o-mini-class equivalent) on that data. The result is near-frontier quality at roughly 10x lower inference cost. For high-volume products where inference cost is a significant line item, this changes the build decision from “marginal improvement” to “structural cost reduction.” PMs who know this pattern have a concrete answer to the “how would you reduce LLM costs” question in design exercises.

The go/no-go checklist

Before recommending fine-tuning in a design exercise or build-vs-buy discussion, clear three gates:

1. Prompt engineering and RAG have been tried and failed a measurable eval. Fine-tuning is the third option, not the first. If you have not established a baseline and run evals, you are guessing.
2. The task is stable and narrow, with 500 to 10,000 labeled examples available. Fine-tuning needs a well-defined, consistent target. Broad generalist tasks do not fine-tune well.
3. Volume is high enough that accuracy gains or cost savings produce positive ROI before the next major model release. In 2026, that window is 4 to 6 months. Model economics that close in 8 months may not survive one release cycle.

If any gate fails, stay with prompt engineering or RAG until it does.

Interview expectations

For the full decision framework across all three options, see RAG vs fine-tuning and choosing between RAG, fine-tuning, and prompting. For the cost side of the model, see LLM unit economics for PMs.