Key Concepts
Ground Truth Generation
Since there's no production traffic yet, an LLM generates realistic student questions straight from the documents themselves. That question → source-document pairing becomes the answer key every metric below is measured against.
Hit Rate & MRR
Hit Rate checks whether the correct document shows up anywhere in the results; MRR goes further and rewards ranking it higher. Together they turn "the search feels okay" into an actual number you can optimize against.
Text Search vs Vector Search
Same 360 questions, two retrieval methods, evaluated side by side. Keyword search actually outperformed embeddings on this dataset — a reminder that "more advanced" doesn't automatically mean "wins."
Hybrid Search — Reciprocal Rank Fusion
RRF fuses two ranked lists using only rank position, sidestepping the problem of comparing BM25 scores to cosine similarities directly. Its damping constant k controls how sensitive the fusion stays to exact rank.
Debugging an Embedding Mismatch
Vector search kept returning results that matched nothing expected. The cause: documents and queries were embedded with two different models — cosine similarity only means something inside one shared embedding space.
Evaluation as a Continuous Process
Every prompt tweak, model swap, or parameter change needs its own evaluation run, not a one-time check before launch. Frameworks like Ragas, DeepEval, and TruLens extend the same idea into production monitoring.
Reciprocal Rank Fusion: Small k vs Large k
Small k (k=1)
rank 1 → 1/(1+1) = 0.500
rank 2 → 1/(1+2) = 0.333 ↓ 33%
rank 3 → 1/(1+3) = 0.250 ↓ 25%
Highly sensitive to
exact rank position
Large k (k=200)
rank 1 → 1/(200+1) = 0.00498 rank 2 → 1/(200+2) = 0.00497 ↓ 0.2% rank 3 → 1/(200+3) = 0.00495 ↓ 0.4% Rank position is nearly flattened out