Most businesses that try ChatGPT on their own work hit the same wall within a day: it writes fluently, but it doesn't know your policies, your contracts or your product catalogue — and when it doesn't know, it guesses. Retrieval-augmented generation (RAG) is the standard fix.
We use RAG in client projects and in our own products, including MyLegalWare, a legal research platform for trade law practitioners, and ClaimResolveAI, which reads health insurance rejection letters. This guide is what we'd explain to a client in the first meeting: what RAG is, how it's built, what it costs to run and where projects go wrong.
What RAG is, and why businesses use it
A large language model (LLM) such as GPT-4o, Claude or Gemini learned from public text. It has never seen your internal documents. RAG closes that gap without retraining the model: when someone asks a question, the system first retrieves the most relevant passages from your documents, then asks the model to generate an answer using only those passages.
That design gives businesses four things a plain chatbot can't:
- Answers from your own content — policies, manuals, contracts, product data, regulations.
- Citations — every answer can point to the document and passage it came from, so people can check it.
- Always current — add or remove a document and the next answer reflects it. No retraining.
- Control — you decide exactly what the AI can know, and who can see which documents.
How a RAG pipeline works, step by step
Every RAG system has two halves: preparing your documents once (indexing), and answering questions (retrieval and generation).
1. Ingest and extract text
Documents come in as PDFs, Word files, spreadsheets, web pages or database records. The first job is getting clean text out. Digital PDFs are easy; scanned letters and photographed pages need OCR (optical character recognition). Tables, headers and footers need care, or they turn into noise.
2. Split into chunks
Long documents are split into smaller passages, typically a few paragraphs each. Chunk size matters more than most people expect: too small and a passage loses its context; too large and retrieval pulls in irrelevant text. Good chunking follows the document's own structure — sections, clauses, numbered points — rather than cutting every N characters.
3. Create embeddings and store them
Each chunk is converted into an embedding — a list of numbers that represents its meaning — and stored in a vector database such as Pinecone, Weaviate or ChromaDB, along with metadata like the document name, section, date and who is allowed to see it.
4. Retrieve the most relevant passages
When a user asks a question, the question is embedded the same way and the database returns the closest passages. Production systems often combine this semantic search with keyword search, filter by metadata (for example, "only this client's documents" or "only the current policy version"), and re-rank the results before passing them on.
5. Generate a grounded answer
The retrieved passages and the question go to the LLM with instructions to answer only from that context, cite sources, and say so when the answer isn't there. The response comes back with references the user can click to verify.
You don't always need the most complex pipeline on day one. One of our own products started with straightforward keyword retrieval over a curated library of regulations rather than a vector database, because the library was small and well structured. Start with the simplest retrieval that finds the right passage, measure it, and add semantic search and re-ranking where the tests show gaps.
Where RAG works well (and where it doesn't)
RAG is at its best when the answer exists somewhere in your documents and people currently spend time finding it.
| Use case | Typical documents | Who benefits |
|---|---|---|
| Internal knowledge assistant | HR policies, SOPs, IT guides, onboarding material | Employees, HR and IT teams |
| Customer support chatbot | FAQs, help articles, product manuals, returns policies | Customers and support staff |
| Legal and regulatory research | Regulations, circulars, case findings, contracts | Lawyers, compliance and audit teams |
| Insurance and claims | Policy wordings, rejection letters, regulator circulars | Policyholders, claims teams |
| Research and reporting | Reports, PDFs, CSV exports | Analysts and management |
RAG is the wrong tool when the answer isn't written down anywhere, when the task is mostly calculation over live data (a database query or a dashboard is better), or when you need the model to take multi-step actions — that's where AI agents and workflow automation come in.
Choosing models, vector databases and hosting
The language model
We regularly work with GPT-4o, Claude, Gemini and open-source models such as Mistral and Llama. The right choice depends on answer quality in your domain, cost per query, speed, and — often the deciding factor — where your data is allowed to go.
The retrieval layer
Managed vector databases (like Pinecone) reduce operational work; self-hosted options (like ChromaDB or Weaviate) keep everything on your infrastructure. Frameworks such as LangChain or LlamaIndex speed up the plumbing, but the quality work — chunking, metadata, evaluation — is specific to your documents.
Hosting and data privacy
- Cloud-hosted: the app and index run on AWS, Azure or Google Cloud, and queries go to an LLM API under the provider's API data terms.
- Your own cloud account: everything deploys inside your AWS or Azure account, so you own the infrastructure.
- On-premise: open-source models on your own servers, with no external API calls at all — the usual choice for highly sensitive documents.
We cover these options in more detail in our data privacy FAQ.
What makes RAG answers reliable
A demo that answers five hand-picked questions proves very little. These are the things that separate a reliable system from a risky one:
- A test set of real questions. Collect 50–100 questions your users actually ask, with the correct answer and source, and run them after every change.
- Citations on every answer. If users can't check the source, they either trust it blindly or stop using it.
- Permission to say "I don't know". When retrieval confidence is low, the system should say it couldn't find the answer, or route to a human — not improvise.
- Grounding rules in the prompt. Tell the model to use only the supplied passages and never invent clause numbers, figures or names.
- Logging and review. Log unanswered and low-confidence questions, review them weekly, and fix the gaps in your documents.
- Access control. Retrieval must respect who is allowed to see which documents, or the AI becomes a way around your permissions.
In ClaimResolveAI, the policy checker is instructed to quote only wording that actually appears in the uploaded policy, and to return "unclear" when the relevant clause isn't in the text. Users trust a system more when it admits what it can't see.
Timeline and running cost
From our projects, a focused RAG layer on an existing application typically takes 3–4 weeks; a complete custom AI chatbot with ingestion, chat interface and admin panel takes 6–8 weeks; a multi-module platform such as a legal research system can take 10–12 weeks.
After launch, the main ongoing cost is LLM API usage, which is billed per token. As a rough guide, 100–500 queries a day typically costs around $20–$80 a month in API fees, and 1,000–5,000 queries a day around $100–$400 a month. Caching and routing simple questions to cheaper models keep high volumes affordable. Hosting and the vector database are the other running costs. For a breakdown of build cost drivers, see how much a custom AI chatbot costs.
Common RAG mistakes
- Indexing everything. Outdated drafts and duplicate versions produce confidently wrong answers. Curate first.
- Ignoring scanned documents. If OCR quality is poor, retrieval never finds the right passage, however good the model is.
- Fixed-size chunking of structured documents. Cutting through the middle of a clause or table separates the rule from its exceptions.
- No evaluation. Without a test set, every prompt tweak is a guess.
- Reaching for fine-tuning to add knowledge. Fine-tuning changes behaviour, not reliable recall — see RAG vs fine-tuning.
- Treating launch as the finish line. Documents change. Budget time to review logs and refresh the index.
Where to go from here
If your first use case is customer- or employee-facing chat, our custom AI chatbot development guide walks through scoping and launch. If you work with regulations or legal documents, read AI for legal and compliance work in India. And if you'd like a second opinion on your own documents, talk to our team — we sign NDAs before any documents are shared.