The question usually arrives as "Can we train ChatGPT on our documents?" The honest answer is that you almost never want to train it on them. You want it to look them up. That's the difference between fine-tuning and retrieval-augmented generation (RAG).
The short answer
- RAG changes what the model knows at answer time. It retrieves relevant passages from your documents for each question, and the model answers from them — with citations.
- Fine-tuning changes how the model behaves. It trains the model further on examples so it adopts a style, format or specialised vocabulary.
If your goal is "answer questions from our policies, contracts, manuals or catalogue", start with RAG. If your goal is "always respond in this exact format or voice, and prompting isn't enough", consider fine-tuning.
RAG vs fine-tuning, side by side
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | The information given to the model for each question | The model's weights — its learned behaviour |
| Best for | Answering from documents and data that change | Consistent style, format, tone or domain vocabulary |
| Citations | Yes — answers point to source passages | No — the model can't say where a fact came from |
| Updating content | Add or remove a document; next answer reflects it | Prepare new training data and retrain |
| Removing information | Delete it from the index — instant | Hard; the model may retain it |
| Access control | Filter what each user can retrieve | Everything trained in is available to everyone |
| Upfront effort | Document ingestion, chunking, index setup | Curating hundreds or thousands of quality examples, plus training |
| Main risk | Poor retrieval → answer misses the right passage | Confident answers with no source; stale knowledge |
Why most business document AI should start with RAG
Business documents change. Policies get revised, prices update, new circulars arrive. With RAG, keeping the AI current is an upload. With fine-tuning, it's a retraining project — and you still can't prove where an answer came from.
Traceability matters even more in regulated work. In our own legal and insurance products, MyLegalWare and ClaimResolveAI, every useful answer has to point back to a Trade Notice, a policy clause or a regulation. That is a retrieval problem, not a training problem. We explain the full pipeline in our RAG development guide.
When fine-tuning is worth it
- Strict output formats the model keeps getting wrong despite clear instructions and examples.
- A distinctive house style across thousands of generated messages.
- Specialised terminology the base model consistently misunderstands.
- Cost at scale — a fine-tuned smaller model can sometimes replace a larger, pricier one for a narrow, high-volume task.
Try better prompts, examples in the prompt, and structured output settings first. Fine-tune only when you can measure the gap those don't close.
Combining them — and where agents fit
Mature systems often use both: RAG for knowledge, a light fine-tune for format or tone. AI agents are a third, separate layer — a model that plans steps and calls tools, like checking an order status or updating a CRM record. Agents frequently use RAG to look things up along the way. Add each layer only when the simpler setup demonstrably falls short — see AI chatbots vs AI agents for when that step makes sense.
A quick decision checklist
- Is the answer written in your documents or data? → RAG.
- Do users need to see where the answer came from? → RAG.
- Does the content change monthly or faster? → RAG.
- Is the problem style or format, not facts? → Prompting first, then fine-tuning.
- Does the AI need to take actions in other systems? → An agent, usually with RAG inside.
Planning a chatbot on top of this? Read our custom AI chatbot development guide, or ask us which approach fits your data.