The Core Problem with Classic RAG: Chunks Lose Their Parent Context
If you've already built a Retrieval-Augmented Generation (RAG) system and its accuracy feels stubbornly mediocre despite using a solid embedding model, the culprit is likely not the model itself — it's how documents get split into chunks.
Classic RAG works by breaking documents into small pieces (typically a few hundred tokens each), converting them into vector embeddings, and storing them in a vector database for semantic similarity search. The problem: once a chunk is separated from its parent document, it loses essential context.
Anthropic's canonical example illustrates this well: imagine a knowledge base of SEC filings, and a user asks, 'What was the revenue growth for ACME Corp in Q2 2023?' The relevant chunk might contain only:
The company's revenue grew by 3% over the previous quarter.
This sentence is factually correct but never states which company or which time period it refers to. The retrieval system struggles to match this chunk to the query, and even if it's found, the generating LLM can misinterpret it without additional context.
Contextual Embeddings: Prepending a Context Summary to Each Chunk
Anthropic's solution, called Contextual Retrieval, starts with a technique called Contextual Embeddings. The idea is simple but effective: before a chunk is embedded, prepend a short explanation that situates it within the overall document.
For example, the original chunk:
The company's revenue grew by 3% over the previous quarter.
Becomes the contextualized version:
This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter.
To generate this context automatically across thousands or millions of chunks, Anthropic uses Claude with a prompt that receives the entire document plus one specific chunk, and asks the model to produce a short, succinct context (typically 50–100 tokens) explaining where the chunk sits within the document. This contextual text is then prepended to the chunk before both the embedding step and the BM25 index creation step.
Across Anthropic's experiments spanning multiple knowledge domains (codebases, fiction, ArXiv papers, science papers), Contextual Embeddings alone reduced the top-20-chunk retrieval failure rate by 35% (from 5.7% to 3.7%).
Contextual BM25: Combining Semantic Search with Keyword Search
Embedding models excel at capturing semantic meaning, but they often miss exact matches — things like a specific error code such as 'TS-999' or a rarely-occurring technical term. This is where BM25 (Best Matching 25) comes in: a lexical ranking function built on top of TF-IDF that also accounts for document length and applies a saturation function to term frequency.
Contextual BM25 applies the same principle as Contextual Embeddings: the exact same contextual text prepended before embedding is also prepended before the BM25 index is built. As a result, keyword search becomes more accurate too, since important terms — company names, dates, technical codes — that were previously lost during chunking now persist in the BM25 index.
When results from both methods are combined via rank fusion and deduplication, the retrieval system benefits from the strengths of both semantic search and precise exact-match search.
The Combined Effect and Reranking
Here's a summary of Anthropic's experimental results on top-20-chunk retrieval failure rate:
| Technique | Failure Rate | Reduction |
|---|---|---|
| Baseline (embeddings only) | 5.7% | - |
| Contextual Embeddings | 3.7% | 35% |
| Contextual Embeddings + Contextual BM25 | 2.9% | 49% |
| Contextual Embeddings + BM25 + Reranking | 1.9% | 67% |
The reranking step works by pulling a larger pool of candidate chunks from the initial retrieval (Anthropic used the top 150), then passing them along with the user's query to a reranking model (they tested the Cohere reranker) to score each chunk's relevance, keeping only the top-K (top 20) chunks to pass into the final LLM prompt.
It's this combination — Contextual Embeddings, Contextual BM25, and reranking together — that produces the widely cited 67% reduction in retrieval failure rate, dropping from 5.7% down to just 1.9%. For developers, that translates into far fewer cases where information genuinely exists in the knowledge base but never surfaces in the top-K retrieval results.
Worth noting: reranking adds runtime latency because it's an extra scoring step, even though the reranker scores all chunks in parallel. There's an inherent trade-off between reranking more chunks for better accuracy versus reranking fewer chunks for lower latency and cost — this is worth tuning per use case.
The Cost Trade-off: One LLM Call per Chunk
The most obvious consequence of Contextual Retrieval is the computational cost at indexing time: every chunk requires one LLM call to generate its contextual text, and that call needs to see the entire parent document to produce accurate context. For knowledge bases with millions of chunks, this can get expensive if done naively — resending the whole document repeatedly for every different chunk.
The solution is prompt caching. With prompt caching, the reference document only needs to be loaded into the cache once; every subsequent call for another chunk from the same document can simply reference the already-cached content instead of resending the whole document from scratch. Assuming 800-token chunks, 8k-token documents, 50 tokens of context instructions, and 100 tokens of generated context per chunk, the one-time cost to generate contextualized chunks comes out to roughly $1.02 per million document tokens.
This is a one-time indexing cost, not a per-query runtime cost — so while it adds a step to the pipeline, its impact on long-term operational cost is relatively small compared to the accuracy gains it delivers.
Practical Implementation Considerations
A few implementation details worth thinking through before rolling out Contextual Retrieval in your own RAG system:
- Chunk size and boundaries still matter — experiment with chunk overlap and sizing appropriate to your domain.
- Embedding model choice affects results; Anthropic found Gemini and Voyage embeddings benefited particularly well from this technique.
- Custom contextualizer prompts tailored to your domain (e.g., including a glossary of terms) can outperform the generic prompt.
- Number of chunks passed to the model — top-20 chunks outperformed top-10 or top-5 in their experiments, though this should still be validated for your own use case.
- Always run evals on your own dataset, since performance can vary meaningfully across domains.
Conclusion
If your RAG system is up and running but its accuracy still disappoints, the culprit is likely not an underpowered embedding model — it's chunks losing context when separated from their parent documents. Contextual Retrieval, combining Contextual Embeddings, Contextual BM25, and reranking, is an experimentally validated approach that can dramatically cut retrieval failures, with additional indexing costs kept manageable through prompt caching. For teams serious about building large-scale knowledge bases, this is one of the highest-ROI techniques available to adopt today.
References
- Anthropic Engineering, 'Introducing Contextual Retrieval,' anthropic.com/engineering/contextual-retrieval
- Claude Cookbook, 'Contextual Embeddings Guide,' platform.claude.com
- Together AI Docs, 'How to Implement Contextual RAG from Anthropic,' docs.together.ai
- DataCamp, 'Contextual Retrieval Anthropic Tutorial,' datacamp.com
