### Hybrid Search: Semantic + Keyword Semantic search (vector similarity) is good at understanding meaning. "How do I cancel my subscription?" matches "Steps to terminate your plan" even though they share no words. But it misses exact matches. "Error code E-4021" might not match a chunk containing "E-4021" if the embedding model treats it as noise. Keyword search (BM25) is the opposite. It excels at exact matches. "E-4021" matches perfectly. But "cancel my subscription" returns zero results if the document says "terminate your plan." Hybrid search runs both, then merges the results. **BM25** (Best Matching 25) is the standard keyword search algorithm. It has been the backbone of search engines since the 1990s. The formula: ``` BM25(q, d) = sum over terms t in q: IDF(t) * (tf(t,d) * (k1 + 1)) / (tf(t,d) + k1 * (1 - b + b * |d| / avgdl)) ``` Where tf(t,d) is the term frequency of t in document d, IDF(t) is the inverse document frequency, |d| is the document length, avgdl is the average document length, k1 controls term frequency saturation (default 1.2), and b controls length normalization (default 0.75). In plain terms: BM25 scores documents higher when they contain query terms (especially rare ones), but with diminishing returns for repeated terms. A document with the word "revenue" 50 times is not 50x more relevant than one with it once. ### Reciprocal Rank Fusion (RRF) You have two ranked lists: one from vector search, one from BM25. How do you combine them? Reciprocal Rank Fusion is the standard approach. ``` RRF_score(d) = sum over rankings R: 1 / (k + rank_R(d)) ``` Where k is a constant (typically 60) that prevents the top-ranked result from dominating. A document ranked #1 in vector search and #5 in BM25 gets: 1/(60+1) + 1/(60+5) = 0.0164 + 0.0154 = 0.0318 A document ranked #3 in vector search and #2 in BM25 gets: 1/(60+3) + 1/(60+2) = 0.0159 + 0.0161 = 0.0320 RRF naturally balances the two signals. A document that ranks highly in both lists gets the best score. A document that ranks #1 in one list but is absent from the other gets a moderate score. This is robust because it uses ranks, not raw scores, so differences in score distributions between the two systems do not matter. ### Reranking Retrieval (whether vector, keyword, or hybrid) is fast but imprecise. It uses bi-encoders: the query and each document are embedded independently, then compared. The embeddings are computed once and cached. This scales to millions of documents. Reranking uses cross-encoders: the query and a candidate document are fed together into a model that outputs a relevance score. The model sees both texts simultaneously and can capture fine-grained interactions between them. A cross-encoder can understand that "What were Q3 earnings?" is highly relevant to a chunk containing "$47.2M in Q3" even if a bi-encoder missed the connection. The trade-off: cross-encoders are 100-1000x slower than bi-encoders because they process the query-document pair jointly. You cannot pre-compute cross-encoder scores for a million documents. The solution: retrieve a larger candidate set (top-50 from hybrid search), then rerank with a cross-encoder to get the final top-5. ```mermaid graph LR Q["Query"] --> H["Hybrid Search"] H --> C50["Top 50 candidates"] C50 --> RR["Cross-Encoder Reranker"] RR --> C5["Top 5 final results"] C5 --> P["Build prompt"] P --> LLM["Generate answer"] ``` Common reranking models (2026 lineup): - Cohere Rerank 3.5: managed API, multilingual, best recall gain on mixed corpora - Voyage rerank-2.5: managed API, lowest latency of the hosted options - Jina-Reranker-v2 Multilingual: open-weight, 100+ languages - bge-reranker-v2-m3: open-weight, strong baseline - cross-encoder/ms-marco-MiniLM-L-6-v2: open-weight, runs on CPU for prototyping - ColBERTv2 / Jina-ColBERT-v2: late-interaction multi-vector rerankers — O(tokens) not O(docs) at scoring time ### Query Transformation Sometimes the problem is not retrieval but the query itself. "What was that thing about the new policy change?" is a terrible search query. It contains no specific terms. The embedding is vague. No retrieval system can find the right documents from this. **Query rewriting**: rephrase the user's query into a better search query. An LLM can do this: ``` User: "What was that thing about the new policy change?" Rewritten: "Recent policy changes and updates" ``` **HyDE (Hypothetical Document Embeddings)**: instead of searching with the query, generate a hypothetical answer, embed that, and search for similar real documents. ``` Query: "What is the refund policy for enterprise?" Hypothetical answer: "Enterprise customers are eligible for a full refund within 60 days of purchase. Refunds are pro-rated based on the remaining subscription period and processed within 5-7 business days." ``` Embed the hypothetical answer and search for real documents similar to it. The intuition: the hypothetical answer lives closer in embedding space to the real answer than the original question does. Questions and answers have different linguistic structures. By generating a hypothetical answer, you bridge the gap between "question space" and "answer space" in the embedding. HyDE adds one LLM call before retrieval. This increases latency by 500-2000ms. Worth it when retrieval quality is poor on raw queries. ### Parent-Child Chunking Standard chunking forces a trade-off: small chunks for precise retrieval, large chunks for sufficient context. Parent-child chunking eliminates this trade-off. Index small chunks (128 tokens) for retrieval. When a small chunk is retrieved, return its parent chunk (512 tokens) for the prompt. The small chunk matches the query precisely. The parent chunk provides enough context for the LLM to generate a good answer. ```mermaid graph TD P["Parent chunk (512 tokens)
Full section about refund policy"] C1["Child chunk (128 tokens)
Standard plan: 30-day refund"] C2["Child chunk (128 tokens)
Enterprise: 60-day pro-rated"] C3["Child chunk (128 tokens)
Processing time: 5-7 days"] C4["Child chunk (128 tokens)
How to submit a request"] P --> C1 P --> C2 P --> C3 P --> C4 Q["Query: enterprise refund?"] -.->|"matches child"| C2 C2 -.->|"return parent"| P ``` The query "enterprise refund?" matches child chunk C2 precisely. But the prompt receives the full parent chunk P, which includes the surrounding context about processing time and submission process. ### Metadata Filtering Before running vector search, filter the corpus by metadata: date, source, category, author, language. This reduces the search space and prevents irrelevant results. "What changed in the security policy last month?" should only search documents from the last 30 days in the security category. Without metadata filtering, you search the entire corpus and might retrieve a 2-year-old security document that happens to be semantically similar. Production RAG systems store metadata alongside each chunk: source document, creation date, category, author, version. Vector databases support pre-filtering by metadata before similarity search, which is critical for performance at scale. ### Evaluation You built a RAG system. How do you know if it works? Three metrics: **Retrieval relevance (Recall@k)**: for a set of test questions with known relevant documents, what percentage of relevant documents appear in the top-k results? If the answer to a question is in chunk #47, does chunk #47 appear in the top-5? **Faithfulness**: is the generated answer grounded in the retrieved documents? If the retrieved chunks say "60-day refund window" and the model says "90-day refund window," that is a faithfulness failure. The model hallucinated despite having the correct context. **Answer correctness**: does the generated answer match the expected answer? This is the end-to-end metric. It combines retrieval quality and generation quality. A simple faithfulness check: take each claim in the generated answer and verify it appears (in substance) in the retrieved chunks. If the answer contains a fact not in any retrieved chunk, it is likely hallucinated. ```mermaid graph TD subgraph "Evaluation Framework" Q["Test questions
+ expected answers
+ relevant doc IDs"] Q --> Ret["Retrieval evaluation
Recall@k: are right
docs retrieved?"] Q --> Faith["Faithfulness evaluation
Is answer grounded
in retrieved docs?"] Q --> Correct["Correctness evaluation
Does answer match
expected answer?"] end ```