How to build a RAG pipeline that works in production, not just in the demo

A RAG demo takes an afternoon. Load a few PDFs, embed them, ask a question, get a good answer. Then real users arrive with real questions, and the answers start going wrong in ways the demo never showed.

Retrieval-augmented generation (RAG) is simple in principle: search your documents, give the relevant passages to the model, let it answer from them. In production, almost every failure happens in the search, not in the model. Here's how I build a RAG pipeline that holds up, step by step.

The RAG pipeline at a glance

Ingest: get documents out of wherever they live and turn them into clean text.
Chunk: split them into passages the right size for search.
Index: create embeddings and a keyword index.
Retrieve: find candidate passages for each question with hybrid search.
Rerank: put the best candidates first.
Generate: answer from the passages, with citations.
Evaluate: measure all of the above on real questions.

If you're not sure RAG is the right tool at all, start with RAG vs fine-tuning.

1. Ingestion: garbage in, garbage out

Most of the quality is decided here. PDFs are the usual trouble: multi-column layouts, headers and footers repeated on every page, tables that turn into word soup, scanned pages that need OCR.

What I do:

Use a parser suited to each format, and look at the extracted text before indexing anything.
Strip repeated headers, footers and page numbers.
Keep structure: headings, section numbers, list items. They help chunking and citations later.
Turn tables into text that still says which value belongs to which row and column.
Store metadata with every document: source, title, date, owner, language, and who is allowed to see it.

2. Chunking: split by meaning, not by character count

Chunks that are too large bury the answer in noise. Chunks that are too small lose the context that makes them make sense.

Split along the document's own structure first: sections, then paragraphs.
Keep a little overlap between neighbouring chunks so a sentence cut at a boundary isn't lost.
Prefix each chunk with its document title and section heading. A chunk that says "the fee is waived after 30 days" is useless if the search doesn't know which product it's about.

There's no universal right size. Test a few on your evaluation set and keep what retrieves best.

3. Hybrid search: embeddings plus keywords

Vector search (embeddings) finds passages with similar meaning even when the words differ. It's weak on exact terms: product codes, names, error messages, legal article numbers. Keyword search (like BM25) is the opposite.

Use both. Run the question through vector and keyword search, then merge the results, for example with reciprocal rank fusion. In my experience this single change fixes more bad answers than switching embedding models.

Add metadata filters too: language, document type, date range, and above all permissions.

4. Reranking: put the best passage first

First-stage search is built for speed and returns a rough list. A reranker, usually a cross-encoder model, reads the question and each candidate together and scores how well they match. Retrieve a few dozen candidates, rerank them, and keep only the top handful for the prompt.

This improves answers and cuts cost, because you send the model fewer, better passages.

5. Generation: answer from the sources, and cite them

The prompt should tell the model to:

answer only from the supplied passages,
say clearly when the passages don't contain the answer, instead of guessing,
cite which passage each claim comes from.

Citations matter more than they seem. They let users check an answer in one click, and they let you debug: when an answer is wrong, you see immediately whether the search brought the wrong passage or the model misread a good one.

6. Evaluation: the step everyone skips

Build an evaluation set before you tune anything: a few hundred real questions, each with the passages that should be retrieved and what a good answer contains. Take them from real tickets, emails or search logs, not from questions you invented.

Then measure retrieval and generation separately:

Retrieval: is the right passage in the top results?
Answer quality: is the answer correct, supported by the cited passages, and does it admit when it doesn't know?

An LLM can help grade answers at scale, but check a sample of its grades by hand. Every change, whether chunk size, embedding model, reranker or prompt, gets measured against this set. Otherwise you're tuning by feel.

7. Permissions and freshness

Two things demos never deal with:

Permissions. Filter retrieval by what the current user may see, at search time. Never rely on the model to hold back something it has already been given.
Freshness. Re-ingest documents when they change and delete removed ones from the index. An assistant quoting last year's price list is worse than no assistant.

Common failure modes

The answer is in a table the parser mangled. Fix ingestion, not the prompt.
The question uses different words than the documents. Hybrid search, plus query rewriting where needed.
Several near-identical document versions. Deduplicate and keep the current one, or filter by date.
The model answers from general knowledge. Tighten the prompt and test that it refuses when the passages don't cover the question.
Questions that need data, not documents. "How many orders did we ship last week?" is a database query, not a RAG question. Route it to a tool that runs the query.

Multilingual RAG: Arabic and French

For multilingual content, check that your embedding model actually handles your languages well. Arabic needs extra care: the same word can appear with or without diacritics, with different forms of alef or hamza, and dialect or Arabizi spellings on top. Normalising text before indexing, and searching by meaning as well as keywords, fixes most of it. French has accents users often skip when typing. I cover the Arabic side in more depth in building an Arabic AI chatbot that understands dialects.

Private RAG

A RAG pipeline can run entirely on your own servers: local embedding model, local index, local LLM. For confidential documents that's often the simplest answer to "where does our data go?". More on that in private AI and on-premise LLMs.

Working together

I'm Driss Briksine, a software and ML engineer based in Morocco. I build AI systems end to end, from data and retrieval to the production software around them. You can see my case studies.

If you have a RAG prototype that works in the demo but not with real users, send me a short brief. I'll tell you where I'd look first.