# Building and Evaluating Advanced RAG: a short course worth taking for the evaluation > The useful lesson in Jerry Liu and Anupam Datta's course is not how to build a pipeline. It is how to find where a RAG system is failing. Original: https://fdetimes.net/en/books-courses/building-evaluating-advanced-rag-course-review/ DeepLearning.AI's *Building and Evaluating Advanced RAG* runs just 2 hours 5 minutes, with 6 videos and 4 code examples. The title puts "building" first. But what is worth taking with you to a customer site is the "evaluating" part: the RAG triad, the three measurements the course uses to score how relevant and how faithful an LLM's answers are. For a Forward Deployed Engineer, that is the part to study most closely. Getting a RAG pipeline running takes an afternoon. The hard part comes when the customer says "the bot is giving wrong answers": you have to show where it goes wrong, why, and whether your fix has made it better. ## Who teaches it, and what does it cover? The two instructors are Jerry Liu of LlamaIndex and Anupam Datta of TruEra/Snowflake. The course was made with two partners, LlamaIndex and TruEra. It is rated Beginner, and there is an extra graded assignment for PRO accounts. The course page shows only an update dated 7 October 2025, not the original release date. That matters when you re-run the notebooks. The code is written on LlamaIndex, so check class names and parameters against the current documentation before you lose an afternoon to version errors. ## Lesson one: the triad is a diagnostic tool TruLens, TruEra's open-source library, designed the triad to catch hallucination on each edge of a RAG architecture. Context relevance checks whether each retrieved piece of context is relevant to the question. Groundedness measures how far the LLM's claims can be traced back to the source text. Answer relevance asks whether the answer is actually useful to the person asking. The real value of the triad is that it tells you which link has failed. Picture a chatbot for looking up HR policy, and an employee asking how many days of annual leave they have left. If the bot answers with the travel-expenses policy, context relevance is probably low: the retriever has pulled the wrong document, and changing the prompt will not help. A second case: the retriever fetches the right leave policy, but the bot gives a number that is not in the document. Groundedness will be low, because the model is overclaiming or inventing details beyond the context it was given. Here the fault lies in answer generation, so the job is to tighten the prompt or change the model, not to touch the retriever. The third case is the hardest to spot: the context is right and every claim is grounded, but the bot only explains the leave-request process in general terms without stating the number of days. The answer is grounded but does not help the person asking, so answer relevance is low. The fix is to instruct the model to answer exactly what was asked. These three numbers turn the complaint "the bot is useless" into a ticket with a specific address. At your first working session with a customer, ask for a few questions the bot has answered wrongly and score the triad for each one before you touch a line of code. ## Lesson two: retrieve small sentences, send the LLM a larger passage Sentence-window retrieval resolves a familiar tension. Small chunks retrieve precisely but lack context. Large chunks carry enough context but tend to drag in irrelevant material. According to the course lectures, the technique retrieves not just the most relevant sentence but a window of the sentences before and after it. In LlamaIndex, the system matches on the embedding of each single sentence. Then `MetadataReplacementNodePostProcessor` replaces each sentence with the window of surrounding sentences before passing it to the LLM. According to the documentation, the default window is 5 sentences on each side, but the code example sets `window_size=3`. So calculate the amount of context from the configuration you are actually running. Do the arithmetic with the default of 5 sentences each side: each matched sentence brings at most 11 sentences into the prompt, and with the top 3 results the LLM reads at most 33 sentences. With `window_size=3`, those figures are 7 and 21; if the matched sentences sit close together, the windows overlap and the real count is lower still. Use this self-calculated ceiling when estimating latency and token costs with the customer. ## Lesson three: high scores are not a guarantee TruLens argues that when all three scores are high, you can be confident the application is not hallucinating. Read that with some scepticism, because other sources treat the triad as a debugging tool rather than a guarantee. The reason lies with the graders themselves: all three metrics are usually scored by an LLM-as-judge. Snowflake has benchmarked those judges. The groundedness judge reached an F1 of 81%, answer relevance 79%, and context relevance only 64%. Agreement between these judges and human graders ranges from fair to high, which means some questions are still scored wrongly, especially on context relevance. Another Snowflake piece shows that improving the judges also requires ground-truth data. They optimised the judges' prompts using Eval-Guided Optimization, and without a benchmark to guide it, TextGrad failed to improve any of the judges. The lesson for FDEs: before reporting triad scores to a customer, have a small set of hand-labelled questions so you know how far the judge drifts. **Key point:** A diagnostic metric tells you where the failure is, but only human grading tells you how far to trust it. ## In what order should you study it? The course is rated Beginner. Even so, you will get more from it if you have already built a basic RAG pipeline yourself, so that by the time you reach the triad you have your own system to score. Watch the sentence-window lesson first, then the evaluation section, and re-run the 4 code examples on your own documents rather than the sample data. On your CV, do not just write that you completed the course. Write one line with before-and-after numbers: for example, that you traced a failure to the retriever and raised context relevance on a set of real questions, and that you checked the judge's scores against a set of labels you graded yourself. A certificate line says only that you finished watching the course. A table of triad scores set beside labels you graded yourself shows an interviewer that you know how to measure, not just how to build. **Try this week:** - Take 20 real questions from a customer (or from a personal RAG project), score all three triad metrics for each one, and mark which questions fail at retrieval and which fail at answer generation - Run sentence-window retrieval with a window of 5 sentences each side, then again with window_size=3, and compare context relevance, groundedness and prompt token count between the two configurations - Hand-grade 10 questions from that set and compare them with the LLM judge's scores, to find out on which metric the judge drifts ## Sources - [Building and Evaluating Advanced RAG - DeepLearning.AI](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/) - [sentence window retrieval - DeepLearning.AI](https://learn.deeplearning.ai/building-evaluating-advanced-rag/lesson/4/sentence-window-retrieval) - [Metadata Replacement + Node Sentence Window - LlamaIndex](https://developers.llamaindex.ai/python/examples/node_postprocessor/metadatareplacementdemo/) - [RAG Triad - TruLens](https://www.trulens.org/getting_started/core_concepts/rag_triad/) - [RAG triad (PromptLayer glossary)](https://www.promptlayer.com/glossary/rag-triad) - [What is the RAG Triad? - TruEra](https://truera.com/ai-quality-education/generative-ai-rags/what-is-the-rag-triad) - [Benchmarking LLM-as-a-Judge for the RAG Triad Metrics](https://www.snowflake.com/en/blog/engineering/benchmarking-LLM-as-a-judge-RAG-triad-metrics/) - [Eval-Guided Optimization of LLM Judges for the RAG Triad](https://www.snowflake.com/en/blog/engineering/eval-guided-optimization-llm-judges-rag-triad/)