Hybrid Search and Reranking - Complete Deep Dive

Prerequisites: Embeddings, Vector Databases, RAG End to End Used in: Advanced Retrieval, Evals, ChatGPT Build it: Lesson 15 - Hybrid Search, RRF, and Reranking implements this as runnable, tested code you can execute offline.

What is Hybrid Search and Reranking?

Hybrid search runs two different retrievers over the same corpus — a lexical one that matches words and a dense one that matches meaning — and merges their ranked lists. Reranking then takes the merged shortlist and re-scores it with a slower, far more accurate model that reads each query-document pair together.

They are two answers to the same complaint: vector-only retrieval returns plausible-looking passages that are not the ones you needed.

Real-world analogy: Hiring. The first pass over a thousand résumés is a keyword filter plus a recruiter’s rough sense of fit — cheap, fast, tuned for not missing anyone good. The second pass is a human reading twenty résumés properly against the actual job description — expensive, slow, tuned for picking the right one. You could not read a thousand résumés properly, and you would not want to hire from the keyword filter alone. Retrieval works the same way, and for the same reason.

How Dense-Only Retrieval Actually Fails

Embeddings compress text into a fixed-size vector of meaning. That compression is the feature, and it is also the failure mode: anything whose value lies in its exact surface form gets blurred.

Read the table as a whole and the design falls out of it. BM25 and dense retrieval do not fail on the same queries. Their error sets barely overlap: lexical fails on vocabulary mismatch, dense fails on exact-form matching. Two retrievers whose weaknesses are correlated would give you nothing by combining them. Two whose weaknesses are complementary give you a union that covers both — which is the entire justification for hybrid search, and the reason it is a bigger win than tuning either retriever alone.

The negation row deserves its own warning. Neither retriever handles it well, and hybrid search does not fix it. That is a query-understanding problem, covered in Advanced Retrieval.

Fusing Two Ranked Lists

Merging is harder than it looks, because BM25 scores and cosine similarities are not on a comparable scale. BM25 is unbounded, corpus-dependent, and query-dependent — a score of 14 means nothing in isolation. Cosine similarity is bounded but distributed in a narrow, model-specific band where 0.80 might be excellent for one embedding model and mediocre for another. Adding them, or comparing them against a shared threshold, is a category error.

Reciprocal Rank Fusion

RRF sidesteps the problem entirely by throwing the scores away and using only rank position.

For each document d:
    RRF(d) = sum over each retriever's list of  1 / (k + rank(d))

rank(d) is d's 1-based position in that list; documents absent from a list contribute nothing.
k is a smoothing constant, conventionally around 60.

Two properties make this the sensible default:

• Scale-free. Rank 1 from BM25 and rank 1 from the dense retriever are directly comparable, because “first” means the same thing in both lists. No normalisation, no per-model calibration, nothing to re-tune when you swap embedding models.
• Agreement is rewarded, dominance is damped. The reciprocal curve is steep at the top and flat further down, so the gap between ranks 1 and 2 is large while the gap between 40 and 41 is negligible. A document ranked moderately well by both retrievers can outrank one ranked first by a single retriever — usually what you want, since single-list dominance is often an artifact.

The k constant controls how much a top rank dominates. Smaller k sharpens the advantage of rank 1; larger k flattens the curve and weights breadth of agreement more heavily. Treat it as a tuning parameter with a sane default, not a magic number.

Score normalisation as the alternative

The other approach is to map both score distributions onto a common range — min-max over the returned candidates, or z-scores — then take a weighted sum: alpha * dense_norm + (1 - alpha) * lexical_norm.

This buys you a genuinely useful thing that RRF cannot give you: a tunable weight, so a corpus of code and identifiers can lean lexical while a corpus of prose leans dense. The cost is fragility. Min-max normalisation depends on the candidate set, so the same document scores differently depending on what else came back. Outliers distort the range. And alpha needs re-tuning whenever either retriever changes.

Start with RRF. Move to weighted normalisation only when you have an evaluation set good enough to prove the weight helps. Several vector stores — Qdrant / Weaviate / OpenSearch / pgvector paired with Postgres full-text — expose one or both natively, which is worth more than implementing fusion yourself.

The Two-Stage Funnel

Fusion improves which candidates come back. Reranking fixes what order they are in. The architecture is a funnel: each stage is cheaper per document and less accurate than the next.

flowchart LR
    Q[User Query] --> B[BM25 over full corpus]
    Q --> D[Dense ANN over full corpus]
    B --> F[Fusion - RRF merge]
    D --> F
    F --> P[Permission filter on metadata]
    P --> R[Cross encoder reranker]
    R --> T[Top 5 for the prompt]
    T --> G[Generator LLM]

    classDef client fill:#f97316,stroke:#c2410c,color:#fff
    classDef service fill:#10b981,stroke:#065f46,color:#fff
    classDef data fill:#fbbf24,stroke:#92400e,color:#000

    class Q client
    class B,D,F,P,R,G service
    class T data

The candidate count narrows sharply at each hop — millions of chunks in the corpus, a few hundred returned by each first-stage retriever, a few dozen surviving fusion and filtering, a handful reaching the prompt. The economics only work because of that narrowing: the expensive model never sees more than a few dozen documents.

Stage 1 is optimised for recall. Its only job is to make sure the right chunk is somewhere in the candidate set. Precision is the reranker’s problem. A first stage that returns the right document at rank 90 has done its job; a first stage that misses it entirely has failed unrecoverably, because no downstream stage can retrieve what stage 1 did not return. Recall at the first stage is the ceiling on the whole system.

Stage 2 is optimised for precision. It reads a shortlist properly and sorts it by genuine relevance.

Bi-Encoder vs Cross-Encoder

This is the mechanism that makes the funnel possible, and it is the question most likely to be asked in an interview.

A bi-encoder — which is what an embedding model is — pushes the query and the document through the encoder independently and compares the two resulting vectors with a cheap distance function.

doc_vec   = encode(document)      # precomputable, done at ingestion
query_vec = encode(query)         # once per request
score     = cosine(query_vec, doc_vec)

Because the document never sees the query during encoding, every document vector can be computed ahead of time and stored in an index. At query time you encode once and do vector arithmetic against millions of precomputed vectors. That is the property that makes corpus-scale search possible at all.

A cross-encoder concatenates the query and the document and pushes the pair through the model together, letting attention run across both:

score = model(query + " [SEP] " + document)   # one forward pass per pair

Now every token of the query can attend to every token of the document. The model can register that this specific error code appears in this specific sentence, that the passage is about the negation of the query’s condition, that the match is on a heading rather than a body mention. That joint reading is why it is dramatically more accurate.

It is also why nothing can be precomputed. The score depends on the pair, and the pair does not exist until the query arrives. Scoring N documents costs N forward passes through a transformer. At corpus scale that is hopeless. On a shortlist of a few dozen, it is a bounded, affordable cost — one batched inference call.

The tradeoff is exact and not a matter of engineering effort: you can precompute or you can read the pair, not both. Two-stage retrieval is the standard resolution — use the precomputable model to get the candidate set small enough that the non-precomputable model becomes affordable.

Between them sit late-interaction models such as the ColBERT family, which store per-token vectors and do a cheaper joint comparison at query time. They land between the two on both accuracy and index cost, and are worth knowing about, but hybrid plus cross-encoder is the path with fewer operational surprises.

Choosing the First-Stage Candidate Count

The knob is how many candidates each first-stage retriever returns before fusion. It trades recall against rerank cost, and the curve has a knee.

• Too small and the right chunk never enters the funnel. This is the unrecoverable failure, and it is invisible unless you measure first-stage recall separately.
• Too large and rerank latency and cost grow linearly while recall barely improves, because you are adding candidates the first stage already ranked as poor.

Measure rather than guess: sweep the candidate count against recall at that count on a labelled query set, and find where the curve flattens. Set the count just past the knee. Then two adjustments. Filtering shrinks the pool, so if permission filtering removes a large fraction of candidates, widen the first stage to compensate — a heavily restricted user otherwise ends up reranking almost nothing. And harder query distributions need wider first stages, so if your traffic skews toward vague conceptual questions, expect the knee to sit further out.

Order matters for cost: filter before you rerank. Reranking documents the user is not entitled to see is money spent to discard results, and running the filter after the reranker is also a leak waiting to happen. See RAG End to End for the access-control argument in full.

Latency Budget

The rerank stage is a real addition to the request path, and it needs a line in the budget rather than being discovered in production.

A hybrid-plus-rerank request spends time on: query embedding, the two first-stage retrievals, fusion, permission filtering, reranking, and generation. Two facts shape the budget. The two first-stage retrievals are independent and must run in parallel — issuing BM25 and the ANN search serially doubles the cheapest part of the pipeline for no reason. And reranking scales with candidate count, so it is the one stage whose cost you directly control, by changing how many candidates you let through.

Generation almost always dominates the total, which is the useful framing: reranking adds a bounded, predictable increment to a path already dominated by token generation. If time to first token is the metric users feel, the fix is usually streaming the answer rather than cutting the reranker. Practical levers when the budget is tight — batch all candidate pairs into one rerank call rather than looping, keep the reranker warm and co-located to avoid cold starts and cross-region hops, cache rerank scores for repeated query-document pairs, and cap candidates by latency rather than by a fixed constant so a slow first stage degrades to a narrower rerank instead of blowing the budget. Full treatment in Latency Engineering, with the caching angle in Prompt and Semantic Caching and Caching.

When Reranking Is Not Worth It

Reranking is usually the best accuracy-per-unit-effort move available, but not always.

• The corpus is tiny. A few hundred chunks means everything relevant already fits in the top results, and often in the context window.
• First-stage precision is already high. If your eval shows the needed chunk is nearly always at rank 1 or 2, there is nothing left to reorder.
• The latency budget is genuinely exhausted. Autocomplete-style interactions measured in tens of milliseconds cannot absorb a transformer pass.
• Queries are overwhelmingly exact-match lookups. BM25 on identifiers is already precise; a semantic reranker adds cost and can even reorder a correct exact match downward.
• You have no evals. Without measurement you cannot tell whether the reranker helped, and you have added a component, a dependency, and a cost line on faith. Build evals first — that ordering holds for every retrieval change on this page.

When to Use

✅ Use when:

• Queries mix natural language with exact identifiers, codes, or error strings
• Retrieval eval shows the right chunk comes back but ranks too low to influence the answer
• The corpus is large enough that the top few dense hits are frequently near-misses
• You need high recall for compliance or support use cases where a miss is expensive
• Users search in multiple languages or with heavy internal jargon

❌ Don’t use when:

• The corpus is small enough that everything relevant already surfaces
• Your latency budget cannot absorb a second model in the path
• Traffic is almost entirely exact-key lookup, where lexical alone is precise
• You have no evaluation set, so you cannot tell whether either change helped

Common Interview Questions

Q1: Why does combining BM25 and dense retrieval help, rather than just picking the better one?

Because they fail on different queries. Lexical search fails on vocabulary mismatch — paraphrases, synonyms, cross-lingual queries. Dense search fails where the exact surface form is the signal — identifiers, error codes, rare proper nouns, quoted phrases. Their error sets are largely disjoint, so the union covers substantially more than either alone. If their weaknesses were correlated, combining them would gain you almost nothing; complementary weakness is the entire premise.

Q2: Explain the difference between a bi-encoder and a cross-encoder and why you need both.

A bi-encoder encodes query and document independently, so document vectors are precomputed at ingestion and a query is one encode plus a vector lookup — that is what makes corpus-scale search possible. A cross-encoder reads the query and document jointly in a single forward pass, so attention runs across both and it is far more accurate, but nothing can be precomputed because the score depends on the pair. That makes it linear in candidate count and unusable over a whole corpus. You use the bi-encoder to cut millions of documents down to a few dozen, then the cross-encoder to order those few dozen properly.

Q3: Why use Reciprocal Rank Fusion instead of just adding the two scores together?

BM25 scores and cosine similarities are not on a comparable scale — BM25 is unbounded and corpus-dependent, cosine sits in a narrow model-specific band — so adding them lets whichever has the larger numeric range dominate arbitrarily. RRF discards the scores and combines by rank position, which is directly comparable across retrievers. It also rewards agreement: a document ranked reasonably by both retrievers can beat one ranked first by only one, which is usually the right call. Score normalisation is the alternative and gives you a tunable weight, at the cost of needing re-calibration whenever either retriever changes.

Q4: How do you pick how many candidates the first stage returns?

Empirically, from a labelled query set. Sweep the candidate count against recall at that count and find where the curve flattens, then set the count just past the knee — beyond it you pay linear rerank cost for negligible recall gain. Two adjustments: widen the first stage if permission filtering removes a large share of candidates, and expect the knee further out if your traffic skews toward vague conceptual queries. The asymmetry matters — a candidate set that is too small loses the right chunk permanently, since no later stage can retrieve what the first stage never returned.

Q5: Reranking pushed your P95 over budget. What do you cut?

First check what actually dominates, because generation is usually the largest term and the reranker a bounded increment on top of it. If the reranker is genuinely the problem, reduce candidate count to the knee of the recall curve rather than removing the stage, batch all pairs into one call instead of looping, keep the model warm and co-located, and cache scores for repeated query-document pairs. Then make the cap latency-driven so a slow first stage narrows the rerank instead of overrunning the budget. If perceived latency is the real complaint, stream the generated answer — that moves time to first token far more than anything in the retrieval path.

Build it in code: Agentic AI Course · Fundamentals: Core Concepts