7-module course covering RAG, vector search, orchestration, evaluation, monitoring, and best practices. Code for all modules lives in the GitHub repo.
Modules
5 of 7 modules completed
Agentic RAG
Built a RAG system from scratch — search, prompt engineering, chunking — then made it agentic with function calling and an LLM-driven loop.
Vector Search
Semantic search with embeddings — moving beyond keyword matching to meaning-based retrieval. Built vector search by hand with numpy, then with minsearch, sqlitesearch, and PGVector; combined keyword and vector search with hybrid RRF fusion.
Orchestration
AI orchestration with Kestra — context engineering, AI Copilot, RAG (static + web search), autonomous agents, and multi-agent delegation, all wrapped in versioned, observable workflows.
Evaluation
Measuring whether a RAG system actually works — generating ground truth with an LLM, scoring retrieval with Hit Rate and MRR, comparing text vs. vector search, and fusing both with Reciprocal Rank Fusion.
Monitoring
Moving from offline evaluation to production observability — instrumenting a RAG pipeline with OpenTelemetry traces and span attributes, persisting them to SQLite, and querying where time, tokens, and cost actually go.
Best Practices
Hybrid search, guardrails, and production patterns for reliable LLM systems.
Project
End-to-end LLM application — putting everything together.