RAG and AI chatbots
RAG or fine-tuning? What your company’s AI assistant actually needs
Most assistants that answer from company documents need retrieval, not a trained model. The difference in plain words, and how to tell which you need.
2 min read
Key takeaways
- Most company assistants need RAG (retrieval at question time), not a trained model.
- RAG keeps answers up to date and shows sources; documents can be added or removed without retraining.
- Fine-tuning teaches a style, format or narrow task; it is a poor way to store facts that change.
- Clean documents, good retrieval, citations and an evaluation set matter more than the model.
When a team says “we want to train an AI on our documents”, they usually mean something simpler, cheaper and easier to keep up to date: an assistant that looks things up in their documents before it answers. That approach is called retrieval-augmented generation, or RAG.
What RAG does
- Your documents (PDFs, docs, wiki pages, help articles) are split into short passages.
- Each passage is turned into a vector, a list of numbers that captures its meaning, and stored in an index.
- When someone asks a question, the most relevant passages are retrieved from that index.
- A large language model writes the answer using only those passages, and links back to them.
Nothing about the model itself changes. It reads your documents at question time, the way a new colleague would check the handbook.
What fine-tuning does
Fine-tuning changes the model’s weights by training it on examples. It’s good at teaching a model a style, a format, or a narrow task it does over and over. It is not a good way to store facts that change, because updating the facts means training again.
Which one you need
Choose RAG when:
- the answers live in documents that change over time
- people need to see where an answer came from
- you want to add or remove documents without retraining anything
Consider fine-tuning when:
- you need a very specific output format or tone, consistently
- the task is narrow and repetitive, with many good examples available
- retrieval alone is already working and you want to refine behaviour
For most company assistants, support bots and internal knowledge search, RAG is the right starting point. Fine-tuning can come later, if it’s needed at all.
What makes a RAG assistant good
The model matters less than people expect. What matters more:
- Clean source documents. Outdated or contradictory pages produce contradictory answers.
- Good retrieval. Passage size, search method and ranking decide whether the right text reaches the model.
- Citations. Every answer should show its sources, so people can check it.
- An evaluation set. A list of real questions with known good answers, re-run after every change, so quality goes up instead of drifting.
If you’re weighing an assistant for your team or your customers, see how I build RAG systems and AI assistants.
