Key Concepts
Offline Evaluation vs. Online Monitoring
Hit Rate, MRR, and an LLM judge against ground truth all work because there's a fixed, known-answer test set. That stops the moment a real user asks a real question — monitoring is what replaces it: capture, store, visualize, act, repeat.
Instrumenting the RAG Pipeline
Subclassed the RAG pipeline instead of rewriting it, overriding just the LLM-calling method to capture latency, prompt/completion tokens, and cost into a dataclass on every call — the pipeline's own logic never changes.
Two Feedback Signals, One Table
Thumbs up/down from real users and relevance labels from an LLM judge land in the same feedback table, distinguished by a source column. Comparing what the judge thinks against what people think is how you'd align the judge over time.
OpenTelemetry Traces & Spans
Wrapped rag(), search(), and llm() each in their own span using OTel — one call becomes a small tree instead of a black box, with search and llm as automatic children of rag simply by being called inside its open span.
Custom Exporter → SQLite
Swapping ConsoleSpanExporter for a hand-written SQLiteSpanExporter changed nothing about which spans get created — only where they land. The instrumentation code never needs to know or care about its destination.
Sampling at Scale
The homework stores and judges every single call — fine for learning, not for production. At real volume, teams sample instead of scoring 100% of traffic and move capture off the request path into an async queue.
Where Does Time Actually Go in a RAG Call?
search span
duration ≈ 0.8 ms
In-memory keyword lookup
No network call
llm span
duration ≈ 3934 ms OpenAI network round trip ≈ 2,000× the search span