Building Agentic RAG with LlamaIndex: keep the thinking, not the code

A question such as “compare the penalty clauses in these three contracts” is where most RAG demos break. Retrieving once and then answering is not enough.

The system has to know when to look up a specific detail, when to summarise and when to read each document in turn. DeepLearning.AI’s short course Building Agentic RAG with LlamaIndex builds exactly that kind of system.

It is taught by Jerry Liu, co-founder and CEO of LlamaIndex, which is the course partner. The course page promises to help you build an agent that can reason over your own documents and answer complex questions.

The course is pitched at beginners: 6 video lessons, 4 code examples and one graded assignment, about 54 minutes in total. The course page lists its most recent update as 7 October 2025.

For FDEs, it is a quick way to share a vocabulary with a client who says “we want an agent that reads our documents”. But the real value of the course lies in the thinking. The code deserves a sceptical eye.

In what order does the course build its ideas?

In the introduction, Liu splits the course into three layers: routing first, then tool use, and finally multi-step reasoning with memory. The hands-on lessons follow that sequence.

You build a router agent that chooses between question answering and summarisation, extend it to pass parameters to tools, and finish with a research agent that handles several documents at once, along with ways to debug and steer it.

The order is worth learning because it is also the order to apply things on a client site. Each layer adds a capability and, with it, another place where things can break. If the router still picks the wrong tool, a multi-step reasoning loop only makes the error bigger.

The router is the cheapest decision layer to measure

Picture a legal client with a repository of contracts. The question “does contract A have a late-delivery penalty clause?” needs the Q&A tool to find a specific passage.

The question “what is contract A about?” needs the summary tool to read the full text. Send the second question to the Q&A tool by mistake and you get back a few disconnected passages and a wrong summary.

The router makes one decision only: which kind of question this is. Because that decision is small, you can score it before discussing a more complex agent.

A hypothetical example: you ask users for 20 real questions and label 12 as Q&A and 8 as summary. The router picks correctly on 17, or 85%. The headline figure says less than where the 3 misses fall.

Suppose all 3 are summary questions sent to Q&A, along the lines of “what are the main points of contract B?”. Then rewrite the descriptions of the two tools to make the boundary clear, and rerun the same 20 questions. Keep the set fixed so that every change tells you whether you moved forward or back.

Tool parameters are a data boundary

The course’s second extension lets the LLM fill in parameters when it calls a tool. On a client site, parameters are usually page numbers, document names or date ranges: in other words, the very scope of data the agent is allowed to read. A question such as “what does page 5 of contract B say?” is answered correctly only if the page parameter is filled in correctly.

The advice is to log every parameter the agent fills in itself. When a client complains of a wrong answer, open the parameter log before suspecting the model, because it is the cheapest thing to rule out.

Debugging and human control

Lesson 4, on the agent’s reasoning loop, is the part most immediately useful for real client work. Liu shows how to step through what the agent is doing and use those very steps to improve it: observability in its most basic form.

He also calls letting users insert guidance into intermediate steps at will a very powerful tool.

With the three-contract comparison, a user might see that the agent has finished reading the first contract and say “skip the appendices, look only at the main clauses”. In a client demo, being able to see each step and cut in like this often reassures people more than a black box that returns only the final answer.

What to drop: the agent-loop code

LlamaIndex’s official documentation now lists AgentRunner and AgentWorker as deprecated in favour of AgentWorkflow and Workflows, and QueryPipeline as deprecated in favour of Workflows. The course page does not say which classes the notebooks use. So open the notebooks and check before copying any agent-loop code into a client project.

The sensible way to learn is to run the notebooks to understand the flow, then rewrite it yourself with the current API. That rewriting exercise is more useful than the course itself, because it forces you to understand each step rather than simply rerun cells.

Who should take it, and in what order?

The course suits engineers who have built basic RAG and are about to take on their first agent project. If you already run agents in production, skim lesson 4 and skip the rest. On Coursera, the course is tagged with the skills Tool Calling, RAG and Debugging.

Take it soon: DeepLearning.AI offers it free only for a limited time, while its learning platform is in beta, so the terms may change. A certificate requires a PRO plan.

The natural next step is Event-Driven Agentic Document Workflows, also from DeepLearning.AI with LlamaIndex, taught by Laurie Voss. It covers event-driven workflows with branching, loops and parallel execution, closer to how LlamaIndex works today.

A repo that rebuilds the router agent on a real document set, with a table of how often the router picks the right tool on a labelled question set and a trace showing a user stepping into the reasoning loop, will demonstrate each of those skills far more convincingly.

The course gives you the framework. Whether it works on real data is for you to prove.