Reranking and Hybrid Search: The Retrieval Layer Everyone Skips

There is a stage in every RAG system that almost nobody touches.

Teams will spend weeks on chunking. They will run bake-offs across embedding models, argue about dimensions, and migrate vector databases. Then they reach the step where the system actually decides which passages the model sees, and they accept whatever the client library does by default: run a cosine similarity search, take the top 5, put them in the prompt.

That default is doing an enormous amount of work, and it is doing it badly. It assumes that a single dense vector comparison is a good enough proxy for relevance, that the top 5 by similarity are the 5 most useful passages, and that the ordering inside those 5 does not matter. All three assumptions are wrong often enough to be the dominant source of retrieval failure in production systems.

I have covered the two layers on either side of this one. Chunking decides what your system is even capable of finding, and your embedding model and vector database decide what "similar" means and how fast you can search it. This article is about what happens in between the moment your index returns candidates and the moment the model receives context: how to search with two complementary methods instead of one, how to combine their results without inventing arbitrary weights, and how a second-stage reranker buys you accuracy that no amount of embedding shopping will.

It is the least glamorous part of the stack and, per hour invested, reliably the highest return.

What the Retrieval Layer Actually Is

The mental model most people carry is a single step: query goes in, chunks come out. The systems that work well have four, and separating them is what makes the whole thing tunable.

Search. One or more index lookups produce candidates. This is where dense vector search lives, and where keyword search should live alongside it.

Fuse. If you searched more than one index, you now have multiple ranked lists that need to become one. This step has a right answer and a lot of wrong ones.

Rerank. A slower, more accurate model rescores the surviving candidates with the query in view. This is the stage almost nobody has.

Select. You decide how many chunks to actually send, in what order, and whether to send any at all. This is a budget decision, not a retrieval one.

The reason to name them separately is that they have different failure modes and different metrics. Stage one either finds the right passage somewhere in its candidate set or it does not. Stage three either ranks it at the top or it does not. Conflating the two is how teams end up swapping embedding models to fix a problem that a reranker would have solved in an afternoon.

Why Pure Vector Search Is Not Enough

Dense retrieval works by turning your query and every chunk into vectors and finding the nearest neighbors. It is genuinely good at what it does: it matches meaning across different wording, handles synonyms and paraphrase for free, and works across languages. Those are real capabilities that keyword search does not have.

But it has structural blind spots, and they are not edge cases. They are the queries your users actually type.

Rare and out-of-vocabulary tokens get blurred. Embedding models compress text into a few hundred or few thousand dimensions. That compression is lossy by design, and what it loses first is exactly what is rarest: product SKUs, error codes, version numbers, internal project names, drug names, legal citations, customer identifiers. A user searching for ERR_CONN_4021 wants the one document containing that exact string. The embedding sees a token soup it has barely encountered in training, produces a vaguely alphanumeric vector, and cheerfully returns five documents about connection errors in general, none of which mention that code.

Semantic similarity is not relevance. These are different things, and the distinction is the root of most retrieval disappointment. A passage that discusses the same topic as the query in the same register will score highly whether or not it contains the answer. Ask "what is the refund window for enterprise plans" and a chunk that talks warmly about refund policy for consumer plans is semantically extremely close and completely wrong. The bi-encoder has no way to notice that the qualifying detail is different, because it never compared the two texts directly. It compared two summaries of them, computed independently, in isolation.

There is no query-document interaction. This is the deep architectural reason for the point above. A bi-encoder embeds the query without knowing the document and embeds the document without knowing the query. All the matching happens afterward, as a dot product between two fixed vectors. Whatever nuance would have come from reading the query and the passage together is simply unavailable, because the model never had both at once.

The ranking inside the top-K is weak. Cosine similarity produces a total order, but the differences between positions 1 and 8 are often noise. Teams treat the ordering as meaningful, then discover it flips when they re-embed with a new model version. Since position in the prompt measurably affects how much the model attends to a passage, a noisy ordering is not a cosmetic problem. It decides which of your retrieved facts the model actually uses.

None of this makes dense retrieval a bad choice. It makes it an incomplete one.

The Other Half: BM25 and What Keyword Search Still Does Best

BM25 is a ranking function from the 1990s. It is still, in 2026, in the retrieval stack of most systems that work well, and it is worth understanding rather than dismissing as legacy.

It scores a document against a query using three intuitions:

Term frequency, with diminishing returns. A document mentioning your search term eight times is more relevant than one mentioning it once, but not eight times more relevant. BM25 saturates: the fifth occurrence adds far less than the second. This is the fix for the keyword-stuffing failure of naive term counting.

Inverse document frequency. A term appearing in almost every document tells you nothing. A term appearing in three documents out of a million is enormously informative. BM25 weights matches by rarity, which means it naturally prioritizes exactly the distinctive tokens that embeddings blur away.

Length normalization. Long documents contain more terms by accident. BM25 discounts matches in long documents so a 40-page manual does not outrank a precise two-paragraph answer just by containing more words.

What you get is a method with a very different profile from dense retrieval. It nails exact matches, rare identifiers, names, quoted phrases, and code symbols. It requires no training, no GPU, no embedding cost, and no re-indexing when a model version changes. Its scores are interpretable: you can point at which terms matched and why.

And it fails in one specific, predictable way: vocabulary mismatch. If the user writes "how do I cancel my subscription" and the document says "terminating a recurring plan," BM25 scores it near zero. There is no lexical overlap, so there is no match, no matter how obviously the passage answers the question.

Now put the two failure modes side by side, because this is the entire argument for what comes next.

The two columns are close to complementary. That is not a coincidence you can ignore. It is a free accuracy improvement sitting in your pipeline, and the only thing standing between you and it is the question of how to merge two ranked lists.

Hybrid Search: Combining Two Rankings Without Making It Up

You run both searches. You get two ranked lists. Now what?

This is the step where most hybrid implementations quietly go wrong, because the obvious approach has a subtle flaw.

The Naive Approach: Weighted Score Fusion

The intuitive move is a weighted sum. Take the dense score, take the BM25 score, combine them with a tunable alpha:

final_score = alpha * dense_score + (1 - alpha) * bm25_score

The problem is that these two numbers do not live in the same universe. Cosine similarity is bounded, typically landing in a narrow band between roughly 0.6 and 0.9 for anything plausible. BM25 is unbounded, scaling with term rarity and query length, and can be 4 for one query and 40 for another. Adding them directly means BM25 dominates whenever the query happens to contain a rare term, and contributes nothing when it does not.

So you normalize, usually min-max within the result set. And that introduces a new problem: the normalization now depends on the result set for this particular query. A query where the top BM25 hit scores 38 and the tenth scores 36 will, after min-max, have its tenth result pushed to zero, even though all ten were near-identical matches. The normalization manufactures a spread that does not exist in the data. Your fusion weights are now tuned against an artifact.

Weighted fusion can work well. It requires per-query score normalization you have thought about, an evaluation set to tune alpha against, and a willingness to re-tune when you change either retriever. That is a real cost, and it buys you something only if you have a genuine reason to weight one channel more heavily.

The Default You Should Actually Use: Reciprocal Rank Fusion

Reciprocal Rank Fusion sidesteps the entire problem by throwing away the scores and keeping only the ranks.

For each document, sum a small contribution from every list it appears in, where the contribution depends on its position in that list:

RRF_score(d) = sum over lists i of  1 / (k + rank_i(d))

with k conventionally set to 60, and rank starting at 1.

The elegance is in what this buys you:

Scale invariance. Ranks are comparable across any two retrievers by construction. There is nothing to normalize, so there is no normalization artifact. You can add a third or fourth retriever tomorrow without retuning anything.

Automatic diminishing returns. The gap between rank 1 and rank 2 contributes far more than the gap between rank 40 and rank 41. That matches how relevance actually behaves at the top of a list, where you care intensely, versus the tail, where you do not.

Agreement is rewarded. A document that appears at rank 3 in both lists accumulates two solid contributions and beats a document sitting at rank 1 in one list and absent from the other. Cross-retriever agreement is a genuine relevance signal, and RRF captures it without you having to model it.

The k parameter mostly controls how sharply top ranks are favored. Lower k makes the top positions dominate; higher k flattens the curve and gives more weight to consensus deeper in the lists. The value 60 comes from the original research and holds up well enough that it is not usually worth tuning. If you do want to bias toward one retriever, add a per-list multiplier rather than abandoning the method.

One practical note on infrastructure: you no longer need to build this yourself. Weaviate, Qdrant, Elasticsearch and OpenSearch, Milvus, Pinecone, and Postgres with pgvector plus a full-text index all support running both channels and fusing them, most of them with RRF available natively. Check whether your database already does this before writing fusion code, and check which fusion it is doing by default, because that detail is often buried in the docs and it materially affects your results.

Reranking: The Stage That Actually Reads the Query

Hybrid search fixes what you find. Reranking fixes what you rank, and it is where the largest single accuracy gain in the retrieval layer usually comes from.

Bi-Encoders and Cross-Encoders

The distinction is the whole story, so it is worth being precise.

A bi-encoder is what your vector database uses. The document is embedded at index time, with no knowledge of any query. The query is embedded at search time, with no knowledge of any document. Relevance is a dot product between two vectors that were computed in mutual ignorance. This is what makes it fast: the expensive part happened during indexing, and search is a nearest-neighbor lookup over precomputed vectors. It is also what makes it imprecise, for exactly the same reason.

A cross-encoder takes the query and one document together, as a single concatenated input, and runs the full model over both. Every token in the query can attend to every token in the document. The output is not an embedding, it is a single relevance score. The model can notice that the query says "enterprise" and the passage says "consumer." It can notice that the passage is about the right topic but answers a different question. That comparison is precisely the thing a bi-encoder architecturally cannot do.

The cost is that nothing can be precomputed. A cross-encoder score exists only for a specific query-document pair, so scoring N candidates means N forward passes at query time. Running a cross-encoder over a million-document corpus is impossible. Running it over 50 candidates that vector search already narrowed down is trivial.

That constraint is what forces the two-stage architecture: a fast, recall-oriented first stage that casts a wide net, and a slow, precision-oriented second stage that sorts what the net caught.

The Reranker Families

For most teams the decision is simple. Start with a hosted cross-encoder reranker if sending your data to a third party is acceptable, or a small open cross-encoder if it is not. Late interaction is worth evaluating when reranking latency becomes your bottleneck and you can afford the storage. LLM-based reranking is worth it when relevance depends on criteria you can express in a sentence but not in a score, for example "prefer passages that state a policy rather than discuss one." It is rarely worth it as a general-purpose ranker, because you pay full generation cost for something a purpose-built model does better and 20 times cheaper.

The last row deserves more attention than it usually gets. Recency, source authority, and document type are often more predictive of usefulness than semantic relevance, particularly in corpora full of superseded versions. A reranker that surfaces the perfect passage from a policy document deprecated 18 months ago has failed, and no amount of model quality fixes that. Blend these signals in after the reranker, or filter on them before the search.

Sizing the Pipeline: The Parameter That Makes It Work

Here is the part that people get wrong when they add a reranker and see no improvement.

A reranker cannot retrieve. It can only reorder what stage one already found. If the correct passage is not in the candidate set, the reranker has no path to it, and your expensive second stage has done nothing but add latency.

This means adding a reranker requires retrieving more, not less. The first stage stops being a precision stage and becomes a recall stage. Its job is no longer "return the 5 best," it is "make sure the right answer is somewhere in here."

Two rules follow directly.

Recall@K of stage one is the hard ceiling on your whole system. If the right chunk is in the top 100 only 82 percent of the time, then 18 percent of your queries are unanswerable no matter how good the reranker, the model, or the prompt is. Measure this number before you buy anything.

Increasing candidates has diminishing returns and linear cost. Going from 10 to 50 candidates usually produces a large recall jump. Going from 100 to 500 usually produces a small one, at five times the reranking latency. Find the knee of that curve on your own data instead of copying someone else's number.

And note what the final selection step is really doing: it is a context budget decision. Sending 20 reranked chunks instead of 5 costs tokens, degrades accuracy through the lost-in-the-middle effect, and slows generation. A good reranker earns its keep partly by letting you send fewer chunks with more confidence, which is a cost saving that shows up directly on your bill. That connection between retrieval quality and token spend is the same one running through why tokens matter.

The Latency Budget

Every argument against reranking is ultimately a latency argument, so it is worth being concrete about where the time goes.

Hybrid search adds little. Two index lookups can run in parallel, so the cost is roughly the slower of the two plus a fusion step that is arithmetic over a few hundred items. On most setups this is single-digit to low tens of milliseconds. If your database does hybrid natively, it may be effectively free.

Reranking adds real time, proportional to candidate count. You are running a model over N pairs. A small self-hosted cross-encoder over 50 candidates on a GPU typically lands in the low hundreds of milliseconds. A hosted API adds network round trip on top. LLM listwise reranking is measured in seconds, not milliseconds. Benchmark it on your infrastructure rather than trusting a number in an article, including this one.

Things that genuinely help:

• Batch the candidates into one call. Scoring 50 pairs in a single batched forward pass is dramatically cheaper than 50 sequential ones. This is the single biggest reranking latency mistake I see.
• Cut the candidate count first. Going from 100 to 50 candidates roughly halves reranker latency. If recall@50 is within a point of recall@100, that is a free win.
• Truncate what you send to the reranker. Cross-encoders have input limits and cost scales with sequence length. Score against a representative window of the chunk rather than a 2,000-token wall of text.
• Cache aggressively on repeated queries. Query traffic in most products is heavily skewed, and the reranked result for a common question is stable. This is a natural companion to the caching strategies in prompt caching and semantic caching, applied one layer earlier in the stack.
• Start streaming the answer. In a chat interface, a few hundred milliseconds of extra retrieval before the first token is far less noticeable than it looks in a trace. Users perceive time-to-first-token, not time-to-retrieval.

The honest framing is a trade: you are spending a few hundred milliseconds to substantially reduce the rate at which your system answers from the wrong passage. For most products that is an obviously good trade. For autocomplete or a sub-100ms search-as-you-type box, it is not, and you should reach for late interaction or skip reranking entirely.

Measure Each Stage Separately, or You Are Guessing

This layer is unusually easy to evaluate, which makes it inexcusable to tune by intuition. The trick is that the two stages need different metrics, and using one metric for both is how teams reach wrong conclusions.

Stage one is measured by recall@K, where K is your candidate count. The only question is whether the right passage made it into the pool. Ordering does not matter here, because the reranker is about to redo it. If recall@50 is low, no reranker will save you and your problem is upstream: chunking, embeddings, or fusion.

Stage two is measured by nDCG@k and MRR, where k is what you actually send to the model. Now ordering is the entire point. nDCG rewards putting relevant passages high and penalizes burying them. MRR tells you how far down the list the first correct answer sits.

The evaluation set is the same asset described in why your demo works but production fails, applied one layer down: 50 to 200 real queries, each labeled with the chunk or chunks that genuinely contain the answer. Build it once, from real user queries rather than invented ones, and every decision in this article becomes an experiment with a number attached instead of an argument in a meeting.

Run the ablation in this order, because it tells you where your actual problem is:

1. Dense only, top 5. This is your baseline and probably your current system.
2. Dense only, top 50, measured on recall@50. This tells you your ceiling.
3. Hybrid with RRF, top 50, recall@50. This tells you what BM25 is adding.
4. Hybrid plus reranker, top 5 after reranking, nDCG@5. This tells you what the reranker is adding.

If step 2 shows that recall@50 is already high while your top-5 precision is poor, you have a ranking problem and a reranker will produce a large, immediate gain. If step 2 shows recall@50 is also poor, stop reading about rerankers and go back to chunking and parsing. This diagnostic takes an afternoon and routinely saves teams a month of aiming at the wrong layer.

How This Stacks With Everything Else

These techniques compose, and the compounding is the reason to care about all of them rather than picking one.

The clearest public evidence is the contextual retrieval evaluation I referenced in the chunking article, measured as top-20 retrieval failure rate. Standard embeddings alone failed 5.7 percent of the time. Adding contextual embeddings brought it to 3.7 percent. Adding BM25 alongside them, which is exactly the hybrid search described above, brought it to 2.9 percent. Adding reranking on top brought it to 1.9 percent, a 67 percent reduction overall.

Read the deltas rather than the totals. Hybrid search and reranking together account for roughly half the total improvement in that progression, and neither of them required changing the embedding model, the chunking strategy, or the LLM. They are additive layers over whatever you already have, which is what makes them such an efficient place to spend engineering time.

Two things worth mentioning that live adjacent to this layer:

Metadata filtering is part of retrieval, not a detail. Filtering by tenant, access level, date range, or document type before search shrinks the candidate space and eliminates a class of catastrophic errors, including the one where a user sees another customer's data. Whether your database filters before or after the vector search matters enormously for both recall and correctness, and it is a property of your specific engine. Go find out which one yours does.

Query transformation is the next layer up. Rewriting a conversational question into a standalone one, expanding it into several sub-queries, or routing it to a different index will all raise stage-one recall further. That belongs to a broader agentic retrieval discussion and is its own article. Get hybrid and reranking working first, because they benefit every query, while query transformation mostly benefits the awkward ones.

When Not to Do This

A guide that only advocates is not useful, so here are the cases where you should skip it.

Your corpus is tiny. A few hundred chunks that comfortably fit in a context window may not need retrieval sophistication, or retrieval at all. That is part of the same RAG versus fine-tuning versus prompting decision about which layer you should be working at.

Your latency budget is genuinely sub-200ms. Search-as-you-type and autocomplete cannot absorb a cross-encoder. Use hybrid search, which is cheap, and consider late interaction rather than a full reranker.

Recall@50 is already poor. Reranking a bad candidate set produces a well-ordered list of wrong answers. Fix the upstream layer first. This is the most common wasted investment in this area.

Your documents never contain identifiers, names, or jargon. If your corpus is uniformly conversational prose in one language and your queries look the same, BM25 will add less. Measure it before adding an index you have to maintain forever.

Your real problem is document parsing. If your PDFs are being ingested with tables scrambled and headers detached from their sections, every layer above is operating on damaged input. That is a different bottleneck, and it deserves fixing first.

Everything else in this article is high-leverage precisely because it addresses failure modes that are otherwise invisible: your system returns confident, topically plausible passages that do not answer the question, and nobody notices until a user does. It is the same class of quiet degradation described in RAG in production, where the system never throws an error, it just gets steadily less useful.

A Practical Checklist

Before you call your retrieval layer production-ready:

• [ ] Both a dense index and a keyword index exist, and both are queried
• [ ] Fusion uses Reciprocal Rank Fusion, or a weighted scheme you have actually tuned on an eval set
• [ ] You know whether your vector database fuses natively and which method it uses by default
• [ ] Stage one retrieves 50 to 100 candidates, not 5
• [ ] A cross-encoder or late-interaction reranker sorts those candidates before generation
• [ ] Reranker calls are batched, not looped one pair at a time
• [ ] The number of chunks sent to the model is a deliberate budget decision, not a library default
• [ ] Metadata filters are applied, and you know whether they run before or after the vector search
• [ ] Recall@K is measured on stage one, and nDCG@k on stage two, on a labeled query set
• [ ] Repeated and high-frequency queries hit a cache instead of the reranker
• [ ] Recency and authority signals are blended in, so superseded documents cannot win

The Right Mental Model

The instinct that leads teams to skip this layer is that retrieval is a solved lookup problem: you have vectors, you find the nearest ones, done. Everything interesting must therefore live in the model.

The better model is that retrieval is a funnel, and each stage has one job. Search casts a wide net, and its only responsibility is that the answer is in the net. Fusion combines evidence from methods that fail differently, so their blind spots do not overlap. Reranking reads the query and the candidates together, which is the only stage that can actually judge relevance rather than approximate it. Selection decides how much of your context budget the result deserves.

Chunking sets the ceiling on what is findable. This layer determines how much of that ceiling you reach. And unlike most improvements in an LLM stack, it does not require a better model, a bigger context window, or a new vendor. It requires adding a keyword index you probably already have access to, fusing two lists correctly, and putting a small model in front of the results.

Most teams are running a two-stage pipeline with the second stage missing and the first stage doing a job it was never designed for. Adding it back is a few days of work, and it usually moves the number further than anything else on your roadmap.

Your embedding model decides what is close. A reranker is the only thing in your stack that decides what is right.

Building production RAG systems? I write regularly about applied AI engineering, retrieval, and the real lessons from production deployments. Find me on LinkedIn or reach out directly at [email protected].