RAG Beyond the Demo: Chunking, Retrieval Quality, and Evaluation

Introduction

Retrieval-augmented generation has a reputation problem: the demo takes an afternoon, the production system takes a quarter, and the difference is almost entirely in retrieval quality. When a RAG system hallucinates or answers "I don't know" to a question whose answer is verbatim in your corpus, the LLM usually isn't the culprit, the retriever handed it the wrong chunks.

This post is about the R in RAG: chunking, hybrid retrieval, reranking, and, most importantly, how to measure whether retrieval is working before you burn weeks tuning prompts.

Chunking: Respect the Document's Structure

The default everywhere is "split every N tokens with overlap." It works, and it's also the first thing to fix, because fixed-size splitting happily cuts tables in half, separates a heading from its section, and splits a code example from the sentence explaining it.

Better: split along the document's own boundaries, then size-limit within them.

def chunk_markdown(doc: str, max_tokens: int = 450, overlap: int = 50):
    sections = split_by_headings(doc)          # ## and ### boundaries
    for section in sections:
        if count_tokens(section.text) <= max_tokens:
            yield Chunk(text=section.text, heading_path=section.path)
        else:
            for piece in split_by_paragraphs(section.text, max_tokens, overlap):
                yield Chunk(text=piece, heading_path=section.path)

Two upgrades with outsized returns:

• Prepend the heading path to every chunk ("Billing > Refunds > Partial refunds: …"). A paragraph that says "this limit is 30 days" is unanswerable without knowing what "this" is; the heading path restores the context the split destroyed. It helps both the embedding and the LLM.
• Keep tables and code blocks atomic. A half-table is noise with high cosine similarity to the question.

Chunk size is a real trade-off, not a formality: small chunks (200-400 tokens) retrieve precisely but lose surrounding context; large chunks (800+) preserve context but dilute the embedding and waste the context window. 300-500 tokens with structure-aware boundaries is a strong default; retrieving a small chunk and then expanding to its parent section at generation time gets you both.

Hybrid Retrieval: Embeddings Miss Exact Matches

Dense (embedding) retrieval is excellent at paraphrase, "how do I get my money back" finds the refunds page. It is surprisingly bad at exact identifiers: error codes, product SKUs, function names, version numbers. A user searching ERR_CONN_RESET_2147 deserves the one document containing that string, and cosine similarity may rank it below ten generically-similar networking pages.

The standard fix is hybrid search: run BM25 (keyword) and vector retrieval in parallel, then fuse. Reciprocal rank fusion needs no tuning and works embarrassingly well:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] += 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

candidates = rrf([bm25_top_n(query, 50), vector_top_n(query, 50)])

Then rerank: take the top 30-50 fused candidates and score each (query, chunk) pair with a cross-encoder reranker, keeping the top 5-8 for the prompt. Rerankers read the query and chunk together, so they catch relevance that bi-encoder embeddings structurally cannot. In every retrieval stack I've tuned, adding a reranker was the single largest quality jump per hour of work.

Measure Retrieval Before You Touch Prompts

You need a labeled set: 50-200 real questions, each mapped to the chunk(s)/document(s) that answer it. Building it is tedious and non-optional, it's the only way to know whether a change helped.

Track three numbers:

• Recall@k: is a correct chunk anywhere in the top k you'll pass to the model? If recall@8 is 60%, your ceiling is 60%; no prompt engineering can answer from context that isn't there.
• MRR (mean reciprocal rank): how high does the first correct chunk rank? Models attend more reliably to context presented earlier.
• Answer faithfulness: does the generated answer actually follow from the retrieved chunks? Judge with an LLM against the labeled set, and audit a sample by hand.

The order of operations this enforces: fix recall (chunking, hybrid, corpus gaps) → fix ranking (reranker) → then tune generation. Teams that start with the prompt are tuning the last 10% while the first 60% is broken.

Generation: Make "Not in Context" a Feature

Two prompt-side rules do most of the anti-hallucination work:

• Instruct the model to answer only from the provided context and to say explicitly when the context doesn't contain the answer, then treat those "not found" responses as retrieval telemetry. Cluster them weekly; they are a map of corpus gaps and chunking failures.
• Require citations: have the model tag which chunk supports each claim ([2]), and render them as links. Citations shift users from "trust the box" to "verify in one click," and unverifiable claims become visible instead of silent.

The Boring Parts That Bite Later

• Index freshness: decide early how updates flow, re-embed changed documents on a schedule or event, and version your index so a bad ingestion can be rolled back like a bad deploy.
• Embedding model migrations: embeddings from different models don't mix in one index. Upgrading means re-embedding the corpus; plan for a blue/green index swap rather than an in-place migration.
• Metadata filters beat similarity: if the user is asking about their account, or about API v3 specifically, a WHERE clause on tenant/version does what no embedding can. Store metadata with every chunk from day one.

Takeaways

• Chunk along document structure, keep heading context attached, keep tables and code atomic.
• Go hybrid (BM25 + dense + RRF) and add a cross-encoder reranker, the highest-leverage hour in RAG.
• Build a labeled retrieval set and track recall@k and MRR; fix retrieval before prompts.
• Make "the context doesn't say" an explicit, logged outcome, and require citations.
• Treat the index like a deployable artifact: versioned, refreshable, rollback-able.

RAG quality is retrieval engineering wearing an AI costume. Measure the R, and the G mostly takes care of itself.