The Evolution of RAG and Retrieval: From Chunking to Agentic Search

Following the complete journey of a retrieval pipeline, from end to end.

0. The Problem at Heart

The fundamental challenge with retrieval-augmented generation (RAG) is simple: models of embeddings work better with short fragments, but a loose fragment loses its context.

Consider this: the phrase "Revenue grew 3%" tells you nothing about which company or which quarter it refers to. To solve this properly, we need to answer two separate questions:

1. Where to cut the document? (Chunking)
2. How to give each piece back the context it lost when cut? (Contextualization)

This is the core tension that drives nearly every evolution in retrieval techniques.

1. Where to Cut: Chunking Strategies

Chunking strategies range from crude to sophisticated, each with different trade-offs. Let's go from coarse to fine:

Strategy	Description	Best For
Fixed-size with overlap	Cut by N characters/tokens with overlap between chunks. Crude but simple.	Quick prototypes, uniform text
Recursive chunking	Try to cut by paragraph first, then sentence, then word. Respects natural boundaries when possible.	General-purpose documents
Sentence-based (with a model)	Use a sentencizer like spaCy that handles abbreviations, decimals, and URLs correctly.	Legal, technical, or messy text
Semantic chunking	Cut where the topic changes by measuring embedding distance between consecutive sentences.	Thematic documents where topics shift naturally
Structural chunking	Respects headings, lists, and tables. For tables, embed each row with its column headers and containing section titles.	Documentation, reports with tables and clear structure
AST-based chunking	For code, cut by functions and classes using tree-sitter.	Source code repositories (as used by Cursor)

2. How to Return Context to Chunks

This is the family that has seen the most innovation recently. All of these solve the same problem, just at different points in the pipeline and with different costs.

The Main Approaches

Technique	When	Needs LLM	Cost at Scale	Context Scope	Notes
Contextual Retrieval (Anthropic, 2024)	At index time	Yes, per chunk	High (mitigated by prompt caching)	Entire document	Prepend LLM-generated context to each chunk for both embedding and BM25.
Late Chunking (Jina, 2024)	At index time	No	Low	Model window (~8K)	Embed the whole document first with long-context model, then average token embeddings per chunk.
Statement Chaining (Wilson Lin)	At index time	No (small classifier)	Very low	Structure + local chain	Uses a DistilBERT classifier to follow dependency chains between sentences, includes only necessary context.
Parent-Document / Small-to-Big	At query time	No	Low	Parent block	Search on small precise chunks, return larger parent blocks to the model.
Sentence Window	At query time	No	Low	N neighboring sentences	Retrieve a sentence, expand with surrounding context on return.
Propositions	At index time	Yes	High	LLM-resolved references	Rewrite to atomic, self-contained assertions with pronouns resolved.
Doc2Query / Hypothetical Questions	At index time	Yes	Medium-High	Generated questions	Generate questions each chunk answers, index those questions instead.

Contextual Retrieval Results

Anthropic's benchmarks showed dramatic improvements when combining techniques:

Configuration	Failure Rate (chunk not in top-20)	Reduction
Base	5.7%	—
Contextual embeddings only	3.7%	35%
+ Contextual BM25	2.9%	49%
+ Reranking	1.9%	67%

Two key insights from Anthropic:

• A generic document summary gave minimal improvement. Specific, chunk-tailored context is what works.
• Much of the total gain comes from BM25 and reranking—standard components, not just contextualization.

3. How to Represent Text

Once chunks are prepared, we need to choose how to encode them for search.

Representation Types

Type	Description	Strengths	Weaknesses
Dense (Bi-encoder)	Single vector per chunk. The standard approach.	Great semantic understanding, fast ANN search	Struggles with exact matches (names, codes, SKUs)
Sparse (BM25/SPLADE)	Term-frequency based or learned sparse vectors.	Excellent for exact matches, interpretable	Less semantic understanding
Late Interaction (ColBERT)	Vector per token, compare each query token to each document token.	Very precise, more nuanced matching	Much more storage space (often 10x+). Different from late chunking!

Compression Techniques

To make vector search scalable, several compression methods are used:

• Matryoshka Representations: Truncate vectors at different dimensions while maintaining performance
• float16: Half-precision, 2x memory savings with minimal loss
• Binary Quantization: 1 bit per dimension, extremely compact, surprisingly effective
• Product Quantization (PQ): Used by CoreNN to fit billions of vectors in RAM

Many modern models like BGE-M3 generate both dense and sparse representations simultaneously—a best-of-both-worlds approach.

4. How to Search

With representations ready, we need efficient search strategies.

Hybrid Search

The modern default is hybrid search: run dense and sparse searches in parallel, then fuse results with Reciprocal Rank Fusion (RRF). This combines semantic understanding with exact-match capabilities.

Approximate Nearest Neighbor (ANN)

For scale, we use ANN indexes:

• HNSW (Hierarchical Navigable Small World): Great for in-memory search, fast and accurate
• DiskANN/Vamana: Optimized for disk-based search, enables much larger corpora

Filtering Strategy

A practical finding: filter after searching is often faster than filtering before. Pre-filtering can constrain the search space in ways that hurt ANN performance, while post-filtering is often more efficient in practice.

5. On the Query Side

The query itself can be optimized before search.

Technique	Description	Effectiveness
Query rewriting	Clean up the query, expand acronyms, fix grammar	Usually helpful, low cost
HyDE (Hypothetical Document Embeddings)	Have an LLM write a hypothetical answer, embed and search with that	Mixed results; Anthropic found limited gains in their tests
Multi-query	Generate multiple reformulations of the question and search all of them	Good for ambiguous or broad queries
Query decomposition	Break complex questions into simpler sub-questions, search each, then synthesize	Great for multi-part questions

6. Reranking

After initial retrieval, reranking reorders results for maximum relevance. This is where the biggest precision gains often come from.

Approach	Description	Trade-offs
Cross-encoder	Reads query and document together, scores relevance directly.	Much more accurate but expensive (can't precompute like bi-encoders)
LLM as reranker	Use an LLM to reorder and filter results.	Very flexible, can do complex reasoning. Wilson serves this via Groq for sub-second latency.
Non-semantic signals	Combine with votes, recency, authority, etc.	In Hackerverse, they combined similarity with post score (social proof) and age penalties.

Going back to Anthropic's numbers: reranking took them from 49% to 67% failure reduction—nearly as much as all the contextualization combined!

7. Global Corpus Structure

Sometimes, top-k retrieval isn't enough. For questions that require understanding the entire corpus, we need global structures.

RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval)

RAPTOR builds a tree by clustering chunks, summarizing each cluster, then clustering those summaries up the tree. This allows retrieval at different levels of granularity—from specific details to high-level themes.

Use case: Questions like "What are all the key arguments made across this 200-page document?"

GraphRAG

GraphRAG takes a different approach:

1. Extract entities and relationships using an LLM
2. Build a knowledge graph
3. Detect communities within the graph
4. Generate summaries for each community
5. Use these community summaries to answer global questions

Use case: Questions like "What are the main themes of this entire corpus?" that no amount of top-k chunks can answer.

8. Agentic Search

The frontier is moving beyond single-shot retrieval to iterative, agentic search. Instead of one search query, the model becomes an active researcher.

The agent follows a loop:

1. Search - Formulate queries based on what it knows
2. Read - Examine returned results
3. Decide - Determine what else it needs to know
4. Repeat - Search again until it has sufficient context

Key optimization: Launch multiple searches in parallel rather than sequentially. As Wilson Lin predicts, agentic search is likely to replace simple retrieval for complex tasks.

9. Code Search in Practice

Different tools take very different approaches to code retrieval. Let's look at how the leading tools solve this.

Cursor: Semantic Index + grep

Cursor builds a sophisticated indexed approach:

• Change detection: Uses a Merkle tree where each file and directory hash is computed from its children. Only changed branches are re-indexed.
• Syntactic chunking: Uses tree-sitter to cut by functions and classes. Embeddings are cached by chunk content.
• Cloud storage: Chunks stored in Turbopuffer, namespaced per codebase.
• Team reuse: New team members compute a simhash of their Merkle tree to find similar indexes and reuse them. Results are verified against the local Merkle tree to prevent cross-contamination.
• Privacy-first: Retrieved chunks are decrypted client-side only.
• Learned embeddings: Trained on actual agent traces—what files did agents open/search before finding the right code? This optimizes for "what the agent actually needs" not "code that's similar".
• Hybrid with regex: For monorepos where ripgrep gets slow, they built a local regex index.

Results: Semantic + grep gives 12.5% better precision than grep alone (6.5%-23.5% depending on model). For repos >1000 files, the improvement in code users keep rises to 2.6%.

Claude Code: No Index, Just Agentic Search

According to Boris Cherny, Claude Code tried a local vector DB early on but switched to iterative agentic search with glob and grep, and it performed better.

• Philosophy: "Glob and grep" wins over RAG
• LSP integration (since Dec 2025): Go to definition, find references, call hierarchy across 11 languages
• Advantage: Always sees live files—no stale index issues after commits
• Trade-off: In multi-million-line monorepos, starting fresh each query can get expensive

Aider: Repo Map with PageRank

Aider takes yet another approach—no embeddings, no grep-centric search:

• Parse everything: Uses tree-sitter to parse the entire repo
• Build a call graph: Maps which definitions reference which
• Rank with PageRank: Custom PageRank biased toward identifiers mentioned in the current chat
• Budget-aware trimming: Cuts the result to fit the token budget
• Output: A structural "map" of the repo rather than raw code chunks

Convergence

The industry is converging on a layered approach:

1. grep/ripgrep - Broad, cheap search
2. LSP - Precise symbol-level navigation and understanding
3. Semantic search - When you don't know the exact name of what you're looking for

10. How to Choose

With so many options, here's a practical decision matrix based on your constraints:

Situation	Recommended Approach
Prose documents, moderate budget	Structural chunking + hybrid search (dense+sparse) with RRF + cross-encoder reranking
Context-critical, small/medium volume	Contextual Retrieval with prompt caching
Massive volume, cost-sensitive	Late chunking or statement chaining
Global questions across entire corpus	RAPTOR or GraphRAG
Code, small/medium repo	Agentic search with grep + LSP
Code, large monorepo	Agentic search + semantic index (Cursor-style)
Many proper names, IDs, or codes	Ensure strong sparse/BM25 component is present

The key insight is that most techniques attack the same problem from different points in the pipeline. You're not choosing one over all others—you're composing the right pieces for your specific constraints of accuracy, latency, cost, and scale.