RAG, Properly: From Naive to Advanced to Modular

Imagine hiring the smartest intern in the world. They have read most of the internet, they write beautifully, and they are wildly confident. There is just one problem: they have never seen a single document from your company. Ask them about your refund policy and they will invent one — fluently.

Retrieval-Augmented Generation is the fix everyone reaches for: before the intern answers, hand them the right pages from the company handbook. Simple idea. The trouble is that "the right pages" is doing an enormous amount of work in that sentence.

I have shipped RAG systems that answered beautifully in the demo and fell apart in week two. This post is everything I wish I had known before the first one — organised the way the research community now thinks about it: naive, advanced, and modular RAG.

RAG is not an LLM problem with some search attached. It is a search problem with an LLM attached.

Part 1 — Naive RAG: The Ten-Line Demo

Naive RAG is the version in every tutorial. It has two phases: an offline indexing pipeline and an online query pipeline.

documents → split into fixed-size chunks → embed each chunk → vector database — Indexing: runs once (or on every document change).

user question → embed question → top-k nearest chunks → stuff into prompt → LLM answer — Querying: runs on every request.

It works surprisingly well on clean, homogeneous documents and short factual questions. Then real users arrive.

Where Naive RAG Breaks

Bad chunks: a table split across two chunks, or an answer that needs paragraph 3 and paragraph 40.
Query–document mismatch: users ask "why was I charged twice?", the doc says "duplicate transaction reversal policy". Embeddings help, but not always.
Exact terms: error codes, SKUs, function names. Dense embeddings are notoriously fuzzy on these.
Noise: top-k returns five chunks, two are relevant, three are confidently misleading.
Lost in the middle: LLMs pay most attention to the start and end of the context. The best chunk in position 3 of 5 can be ignored.
No feedback loop: if retrieval fails, the model answers anyway.

Advanced RAG is a catalogue of fixes for exactly these failures, organised by where they sit in the pipeline.

Part 2 — Advanced RAG: Fix Each Stage

question → pre-retrieval (fix the query + index) → retrieval (find better candidates) → post-retrieval (filter + order) → generation — Advanced RAG keeps the naive pipeline shape but adds an optimisation layer at every stage.

Think of a librarian. A bad one takes your words literally and hands you the first five books with matching titles. A good one asks what you actually mean (pre-retrieval), knows several ways to search the stacks (retrieval), and skims the books before handing you the two that matter (post-retrieval).

Pre-Retrieval: Fix the Index and the Question

1. Chunking Is the Most Underrated Knob

Fixed-size chunks with overlap: the baseline. 300–800 tokens with 10–20% overlap is a sane start.
Structure-aware chunking: split on headings, sections, list items, code blocks. Keep tables whole.
Semantic chunking: split where embedding similarity between neighbouring sentences drops — i.e. where the topic changes.
Small-to-big (parent-document): embed small chunks for precise matching, but return the bigger parent section to the LLM.
Sentence-window: retrieve a single sentence, then expand to N sentences around it before generation.
Contextual chunk headers: prepend "Document: Billing FAQ › Section: Refunds" to each chunk before embedding. Cheap, and it often helps a lot.

Analogy: chunking is how you cut a pizza. Cut it into crumbs and nobody gets a proper slice; leave it whole and nobody can pick it up.

2. Enrich the Index With Metadata

Store source, date, product, language, access level and document type with every chunk. Then filter before you search: "only docs for product X, updated after 2024". Metadata filtering is boring and it fixes a shocking number of wrong answers.

3. Rewrite the Query

Query rewriting: ask a small LLM to turn a chatty, context-dependent question ("and what about for enterprise?") into a standalone search query.
Multi-query: generate 3–5 paraphrases, retrieve for each, merge the results. Covers vocabulary mismatch.
Query decomposition: split "compare plan A and plan B pricing in India" into sub-questions and retrieve for each.
Step-back prompting: ask a more general question first ("how does billing work?") to pull in background context.
HyDE (Hypothetical Document Embeddings): let the LLM write a fake answer, embed that, and search with it. Answers look more like documents than questions do.

raw question → rewrite / multi-query / decompose / HyDE → set of search queries → retrieve for each → merge — Query transformations fan out one question into several better searches.

4. Route the Query

Not every question should hit the same index. A router (rules, a classifier, or an LLM call) decides: product docs vs. API reference vs. SQL database vs. "no retrieval needed, just chat". Routing is the first step towards modular RAG.

Retrieval: Find Better Candidates

Sparse, Dense, and Hybrid

Sparse (BM25): keyword matching with smart weighting. Great for exact terms, IDs and rare words. Blind to synonyms.
Dense (embeddings): matches meaning. Great for paraphrases. Weak on exact identifiers and numbers.
Hybrid: run both and fuse the results. In every production system I have worked on, hybrid beat either one alone.

The simplest robust way to merge two ranked lists is Reciprocal Rank Fusion. It ignores the raw scores (which are not comparable between BM25 and cosine similarity) and only uses rank positions:

RRF(d) = Σ over retrievers r of  1 / (k + rank_r(d)),   k ≈ 60

query → BM25 → ranked list / vector search → ranked list → reciprocal rank fusion → top 50 candidates — Hybrid retrieval: two retrievers with different blind spots, fused by rank.

Other Retrieval Upgrades

Fine-tuned embeddings: train the embedding model on your own (query, relevant doc) pairs. Even a few thousand pairs help on domain jargon.
Multi-vector / late interaction (ColBERT-style): keep one vector per token instead of one per chunk. More precise matching, more storage.
Recursive / hierarchical retrieval: search document summaries first, then chunks within the best documents.
Retrieve more, then cut: fetch top 50–100 cheaply, let the post-retrieval stage pick the final 5.

Post-Retrieval: Filter, Order, Compress

1. Rerank

Your first-stage retriever is a fast bouncer checking IDs at the door. A reranker is the slow, careful interviewer inside. Cross-encoder rerankers read the query and each candidate together and score true relevance. They are too slow to run over millions of chunks, but perfect for re-scoring the top 50.

top 50 candidates → cross-encoder reranker → top 5 → LLM — Two-stage retrieval: cheap recall first, expensive precision second.

2. Compress and Clean

Deduplicate near-identical chunks (very common with versioned docs).
Contextual compression: extract only the sentences relevant to the question from each chunk.
Drop anything below a relevance threshold — returning nothing is better than returning noise.

3. Order for the Model, Not for You

Because of the lost-in-the-middle effect, put the most relevant chunks at the start and the end of the context, weaker ones in the middle. Also label each chunk with its source so the model can cite it.

4. Check Before You Answer

Corrective RAG (CRAG): grade the retrieved documents; if they look bad, rewrite the query or fall back to another source (e.g. web search).
Self-RAG: the model decides when to retrieve and critiques its own output for support and relevance.
Grounding check: after generation, verify each claim is supported by a retrieved chunk. If not, say "I don't know".

Part 3 — Modular RAG: LEGO, Not a Pipeline

Advanced RAG still thinks of the system as one fixed line. Modular RAG treats every technique above as a swappable module — and the flow between them as something you can design, branch and loop.

question → router → search module: hybrid index / SQL / API module / graph module / memory module → fusion + rerank → generate → judge: grounded? (loop: not grounded → rewrite query and search again) — A modular RAG flow. Modules are chosen per query, not hard-wired.

Common Modular Patterns

Rewrite → retrieve → read: the classic, now with a query-rewrite module in front.
Iterative retrieval: generate a partial answer, use it to retrieve again, repeat.
Adaptive retrieval: skip retrieval entirely when the model is confident or the question is small talk.
Agentic RAG: an agent with tools (search, SQL, calculator) decides which retrieval to run and when to stop — ReAct with a search tool.
Graph-based modules: when the answer lives in relationships between entities, not in any single chunk (my next two posts: GraphRAG and LightRAG).

Naive RAG is a recipe. Advanced RAG is a better recipe. Modular RAG is a kitchen.

Part 4 — Evaluate Each Half Separately

When a RAG answer is wrong, there are only two possibilities: retrieval failed to find it, or generation failed to use it. If you only score the final answer, you cannot tell which one to fix.

question → retrieval → retrieval metrics: recall@k, MRR, context precision → generation → generation metrics: faithfulness, answer relevance — Score retrieval and generation separately so you know which knob to turn.

Build a small golden set: 50–200 real questions, each with the document(s) that should answer it.
Retrieval: recall@k (did the right chunk show up?) and MRR (how high?).
Generation: faithfulness (is every claim supported by the context?) and answer relevance (does it actually answer the question?).
Frameworks like RAGAS package these, but hand-labelling 50 examples teaches you more than any dashboard.

My Default Stack, If You Want One

Structure-aware chunks with contextual headers and metadata.
Query rewrite for multi-turn chats.
Hybrid retrieval (BM25 + dense) fused with RRF, top 50.
Cross-encoder rerank to top 5–8.
Order by relevance, cite sources, allow "I don't know".
A golden set and retrieval metrics from day one.

Start naive. Measure. Add exactly one technique at a time and keep only what moves the numbers. Most RAG systems do not need every trick in this post — but every production RAG system needs to know which trick it is missing.