Building Custom RAG Pipelines Using Hybrid Search: Combining BM25 Keyword Scoring and Dense Vector Distance

Vigtige pointer

1. Index for both modalities. Build a BM25 index for lexical matching and a vector index for semantic matching on the same chunks.
2. Normalize scores. BM25 and cosine scores are on different scales; normalize before fusion.
3. Fuse with reciprocal rank fusion. RRF combines ranked lists without score calibration and is robust to scale differences.
4. Tune the fusion weight. A weighted combination of BM25 and vector scores lets you favor precision or recall.
5. Rerank the fused results. A cross-encoder reranker on the top candidates improves final ordering.
6. Handle short queries. BM25 excels on short keyword queries; ensure the hybrid does not dilute exact matches.
7. Evaluate retrieval separately. Measure recall at k and precision at k for both modalities before combining.
8. Cache embeddings. Re-embedding the corpus on every change is expensive; store and reuse vectors.

Architecture

A query is sent to both the BM25 index and the vector index. Each returns a ranked list. Reciprocal rank fusion merges the lists into a single ranking. A cross-encoder reranker then reorders the top results, which are passed to the language model as context.

Production configuration

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np

bm25 = BM25Okapi(tokenized_corpus)
model = SentenceTransformer('all-MiniLM-L6-v2')
vecs = model.encode(corpus)

def hybrid_search(query, k=10):
    bm25_scores = bm25.get_scores(query.split())
    qvec = model.encode([query])[0]
    dense_scores = np.dot(vecs, qvec) / (np.linalg.norm(vecs, axis=1) * np.linalg.norm(qvec))
    # reciprocal rank fusion
    bm25_ranks = np.argsort(bm25_scores)[::-1]
    dense_ranks = np.argsort(dense_scores)[::-1]
    rrf = {}
    for rank, idx in enumerate(bm25_ranks[:k]):
        rrf[idx] = rrf.get(idx, 0) + 1.0 / (60 + rank + 1)
    for rank, idx in enumerate(dense_ranks[:k]):
        rrf[idx] = rrf.get(idx, 0) + 1.0 / (60 + rank + 1)
    return sorted(rrf, key=rrf.get, reverse=True)[:k]

from sentence_transformers import CrossEncoder
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query, candidates, top_n=5):
    pairs = [(query, corpus[i]) for i in candidates]
    scores = reranker.predict(pairs)
    order = np.argsort(scores)[::-1]
    return [candidates[i] for i in order[:top_n]]

def recall_at_k(retrieved, relevant, k=10):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)

# test both modalities
for mode in ['bm25', 'dense', 'hybrid']:
    r = retrieve(mode, queries, k=10)
    print(mode, recall_at_k(r, ground_truth))

Edge cases from production

• BM25 over long documents favors length; chunk consistently to avoid bias.
• Dense models trained on general text underperform on domain jargon; fine-tune embeddings.
• RRF constant 60 is a default; tune it for your corpus size and query distribution.
• Hybrid search doubles index storage and query cost; cache results for repeated queries.

Ofte stillede spørgsmål