Use retrieval when the model needs to know your facts, and fine-tuning when it needs to behave in a set way. Try them in order: prompting first, then retrieval, then fine-tuning, and move to the next step only when the evaluation set shows the current one falls short. Questions about policies, products and client files are knowledge problems, and retrieval solves them with less upkeep than training.
The decision order
Prompting is the instruction, the examples and the output format you send with each request. A change takes minutes, costs nothing to undo and needs no training data. Write the prompt, run it against the evaluation set and record the score. That score is your prompt-only baseline.
Retrieval augmented generation (RAG) fetches the passages relevant to a question from your documents and puts them in the prompt. The method comes from Lewis and others (2020), who paired a language model with a searchable index of documents. In a private build the index is PostgreSQL with pgvector, Qdrant or Elasticsearch, inside your own network. RAG on private infrastructure covers the parts.
Fine-tuning trains the weights of the model further on your own examples and changes the model itself. It is the slowest step to build and the hardest to undo, so it comes last.
What each one fixes
The test is whether the failure is about what the model knows or how it acts.
| Problem | Prompting | Retrieval | Fine-tuning |
|---|---|---|---|
| Model does not know your policies, products or clients | Partly, for a short fixed text | Yes | Poorly |
| Facts change weekly | No | Yes, re-index the document | No, retrain each time |
| Answer must cite a source document | No | Yes | No |
| Users must only see documents they are allowed to see | No | Yes, filter at retrieval | No |
| Output must follow a strict format or label set | Often | No | Yes |
| House style, tone or internal shorthand | Partly | No | Yes |
| A small model must do one narrow task well | Partly | No | Yes |
| Prompt is long and repeated on every call | No | No | Yes, the behaviour moves into the weights |
Research supports the split. Ovadia and others compared the two for adding knowledge. They report that RAG “consistently outperforms” unsupervised fine-tuning, for knowledge seen in training and for new knowledge. Gekhman and others found that a model learns new facts slowly during fine-tuning. When it does learn them, its tendency to hallucinate increases. A fact you want the model to state belongs in a document the model can read and cite.
Access control decides many cases on its own. A retrieval layer can filter passages by the permissions of the user who asks. A fine-tuned model holds what it learned for every user.
Data requirements
Retrieval needs the documents, parsed into clean text with tables intact, split into passages and tagged with metadata: source, date, owner and access group. It also needs a set of questions with known answers and known source passages. With that set you can score retrieval and the answer separately. This is data engineering work, and it needs an owner. Pipelines in Airflow or Dagster keep the index in step with the source systems.
Fine-tuning needs input and output pairs that show the behaviour you want. Quality counts more than volume in this data. The LIMA paper describes a model fine-tuned on “only 1,000 carefully curated prompts and responses”. The right number for your task comes from experiment: train on a subset, measure, add examples and measure again.
Personal information changes the design. The OAIC’s guidance on developing and training generative AI models uses the example of a bank that fine-tunes a chatbot on customer queries. It says the bank “should first consider whether the fine-tuning dataset needs to include the personal information”. It also says de-identification “may be difficult to achieve”. Under APP 11, an entity has obligations to destroy or de-identify personal information in certain circumstances. You can delete a document from an index. You cannot reliably remove one customer from the weights. For insurance and banking data, keep personal information in the retrieval layer and train only on de-identified examples.
LoRA and QLoRA
Full fine-tuning updates every weight in the model, which needs memory for the weights, the gradients and the optimiser state. LoRA and QLoRA reduce that memory.
LoRA (low-rank adaptation), from Hu and others (2021), freezes the original weights and trains two small matrices beside each chosen layer. Their product is the update to that layer. The rank, written r, sets the size of those matrices. For one 4,096 x 4,096 weight matrix, full training updates 4,096 x 4,096 = 16,777,216 values. LoRA at rank 16 trains 2 x 4,096 x 16 = 131,072 values, under 1% of the original. Against GPT-3 175B fine-tuned with Adam, the authors report 10,000 times fewer trainable parameters and 3 times less GPU memory. They also report no added inference latency.
QLoRA, from Dettmers and others (2023), loads the frozen base model in 4-bit precision and trains LoRA adapters on top. It adds a 4-bit NormalFloat data type, double quantisation and paged optimisers to save memory. The paper reports that it “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU”.
A QLoRA setup with the Hugging Face PEFT library looks like this:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, quantization_config=bnb)
config = LoraConfig(
r=16,
lora_alpha=16,
lora_dropout=0.1,
target_modules="all-linear", # every linear layer, as in QLoRA
task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()The PEFT documentation says a higher rank gives the adapter more parameters and more learning capacity. After training, merge_and_unload() folds the adapter into the base weights for serving. You can also keep adapters separate. vLLM serves LoRA adapters “on a per-request basis with minimal overhead” when you start it with --enable-lora and --lora-modules. One base model can then carry a different adapter for each team or task.
Upkeep when documents change
The two approaches age in different ways, and the running cost is where the choice is felt.
| Event | Retrieval | Fine-tuning |
|---|---|---|
| A policy document is updated | Re-parse and re-index that document | No change to behaviour, and old facts stay in the weights |
| A document must be withdrawn | Delete it from the index | Retrain without it, then evaluate again |
| A new document set is added | Ingest it with its access group | Not needed for knowledge |
| Output format changes | Change the prompt | Build new examples and retrain the adapter |
| You move to a newer base model | Re-run the evaluation set | Retrain the adapter, because it matches the old model’s layer shapes |
The last row is the cost most teams miss. Open-weight families release new versions often, and each upgrade means a new training run, a new evaluation and a new sign-off. Keep the training data, the training script and the evaluation set under version control. A retrain is then a repeatable job in MLflow rather than a new project.
Evaluation against two baselines
A fine-tuned model has to beat two things on the same held-out evaluation set: the base model with the same prompt, and the best prompt-only or retrieval system you have. Without both baselines you cannot tell whether the gain came from training or from a better prompt.
- Hold out the evaluation cases before training, and keep every one of them out of the training data.
- Fix the success threshold before you train, for example the minimum gain in field accuracy you will accept for the upkeep cost.
- Score each system on the task metric, on format validity and, for retrieval, on faithfulness to the retrieved passages. Ragas style metrics cover the last one.
- Run a regression set of general tasks to measure whether abilities you still need dropped after training.
- Log every run with the base model version, adapter version and data version.
Evaluation and observability for self-hosted LLMs covers the harness. Choosing an open-weight model covers the base model choice that comes before any of this.
Decision checklist
- The prompt-only baseline is scored on a fixed evaluation set.
- The failures are sorted into knowledge failures and behaviour failures.
- Knowledge failures go to retrieval first, with access control at the retrieval layer.
- Retrieval is scored separately from the answer, to show which part fails.
- Behaviour failures that remain after prompt work have a fine-tuning case.
- Training examples exist, or a person owns writing them, and they contain no personal information you cannot justify under the OAIC guidance.
- The success threshold is written down before training starts.
- Someone owns the retrain when the base model or output format changes.
What to decide next
Score the prompt-only baseline, sort the failures, and build retrieval for the knowledge failures before you plan any training. Bring fine-tuning in only for behaviour that prompting and retrieval did not fix, with the threshold set in advance. Our LLM integration and RAG and model fine-tuning services cover both builds inside your own environment.