Fine-Tuning vs RAG vs Prompt Engineering: Which Does Your AI Project Need?

Most AI projects should start with prompt engineering, add RAG (retrieval-augmented generation) when the model needs facts it was never trained on, such as your prices, policies or product data, and consider fine-tuning only when the model knows enough but behaves inconsistently. The three are not rivals: they fix different problems, and many working systems combine two or all three.

The hard part is knowing which problem you have. This guide explains each technique and the order to try them in. Sources were checked on 14 November 2026.

What is the difference between prompt engineering, RAG and fine-tuning?

All three change what a large language model (LLM) produces, but they act on different things:

• Prompt engineering changes the instructions you send with each request: the role, the rules, the format you want and a few examples. Nothing about the model itself changes.
• RAG changes the information you send with each request. Before the model answers, a search step pulls the most relevant passages from your documents or database and adds them to the prompt, so the answer is grounded in your content.
• Fine-tuning changes the model. You train it further on many examples of the inputs and outputs you want, and it learns that pattern so you do not have to spell it out every time.

OpenAI's guide "Optimizing LLM Accuracy" uses an exam analogy that is worth borrowing. RAG is like letting a student bring the textbook into the exam: the knowledge is right there in context. Fine-tuning is like the student attending class for months: the knowledge or skill becomes learned behavior. The same guide stresses that the two are additive, not exclusive.

When is prompt engineering enough?

More often than people expect. OpenAI's guide calls prompt engineering the starting point and notes that for tasks such as summarization and translation a zero-shot prompt (instructions with no examples) can already reach production-level accuracy.

Prompt engineering is usually enough when:

• The task relies on general knowledge or text the user provides, such as summarizing a call or classifying a ticket.
• The output problems are about clarity: the model is too long-winded, ignores a format, or misses an edge case you never mentioned.
• You can show what "good" looks like with a handful of examples that fit comfortably in the prompt.

It is also the cheapest thing to change: no data pipeline, no training job. Anthropic's prompt engineering overview makes a useful point in the other direction: not every failing test is a prompt problem, and some goals, such as lower latency or cost, are sometimes easier to reach by choosing a different model.

When do you need RAG?

You need RAG when the model is missing facts. A general-purpose model was trained on public data up to a cutoff date. It does not know your current price list, your return policy, last week's inventory or the contents of your internal wiki, and no amount of clever wording will make it know them.

The technique comes from a 2020 research paper by Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," which paired a language model with a searchable store of documents. In business terms, RAG is the right tool when:

• Answers must come from your own documents, contracts, manuals or knowledge base.
• The information changes often. Updating a document is far easier than retraining a model.
• You need to show where an answer came from, for example by linking the source policy page.
• The content is too large to paste into every prompt.

RAG brings its own work: choosing and updating the content and testing that search finds the right passage. When a RAG system gives a wrong answer, the cause is often retrieval (the wrong passage was found) rather than the model. Our earlier guide, What is RAG?, covers how it works in more detail.

When is fine-tuning worth it?

Fine-tuning is for behavior, not facts. OpenAI's guide describes it as the fix when a model produces inconsistent results, uses the wrong format, or does not reliably follow a required way of reasoning, and says its aim is consistency of behavior.

OpenAI's "Model optimization" documentation lists the situations where fine-tuning beats prompting alone. In summary, it helps when you can:

• Provide more example inputs and outputs than would fit in a prompt.
• Use shorter prompts with fewer examples, which lowers the token cost of every request.
• Train on sensitive examples once instead of sending them with each request.
• Get a smaller, cheaper model to handle one narrow task well.

The same page describes the methods on offer: supervised fine-tuning (you supply correct example responses), direct preference optimization (you supply a better and a worse response for each prompt), and fine-tuning for image understanding. Each needs a prepared dataset, usually hundreds of good examples or more, plus time to evaluate the result.

Fine-tuning is a poor fit for teaching a model your changing facts. Training data goes stale the day your prices change, and a fine-tuned model still cannot cite where an answer came from. For that, use RAG.

How do you decide? A simple diagnostic

Before choosing a technique, write down what "good" means and build a small test set. Both OpenAI and Anthropic put this first: OpenAI's model optimization guide starts with writing evals to establish a baseline, and Anthropic's overview assumes you already have success criteria and a way to test against them. Twenty to fifty realistic questions with known good answers is a practical start for a small team.

Then run the test and look at how the answers fail:

1. Wrong or made-up facts about your business. The model lacks knowledge. Add RAG.
2. Right facts, wrong shape. Wrong tone, broken format, skipped steps. Improve the prompt first; if it is still inconsistent after a serious effort, consider fine-tuning.
3. Good answers, but too slow or too expensive at volume. Try a smaller or different model, then consider fine-tuning a smaller model on examples from the larger one.
4. Mixed failures. Fix knowledge gaps with RAG first, then work on behavior.

Re-run the same test after every change so you know whether it helped.

Can you combine them?

Yes. A common setup is a careful prompt plus RAG over company documents, with fine-tuning sometimes added later for a house style or format.

Combining is not automatically better, though. In one case study in OpenAI's accuracy guide, adding RAG on top of a fine-tuned model lowered accuracy on that task. The lesson is to add each layer for a specific reason and measure it, not to stack techniques because they sound advanced.

What should a small business try first?

For a US small or mid-sized business, the order is: a test set, then prompt engineering, then RAG if answers depend on your own or changing information, then fine-tuning only if a measured behavior problem remains or volume makes a smaller tuned model worth the setup.

If you want help working out which of these your project needs, our LLM and AI agent development team builds prompt, RAG and fine-tuning setups, and our AI consulting for small businesses helps you scope the project and the test set before any build starts. We work remotely with clients across the US.