LLM Zoomcamp: a free RAG course that also teaches evaluation and monitoring

A RAG chatbot that can answer a few questions is only the first half of the job. The second half starts with a question a client may ask in week three: “Is it giving the right answers, and how do you know?” LLM Zoomcamp, a free course from DataTalks.Club, stands out because it deals with that question head-on.

Besides the basics of LLMs and RAG, the course has one module on evaluation and another on monitoring. For anyone aiming to become an FDE, these are the parts most worth studying, because an FDE’s work begins where the demo ends.

Who runs the course, and what does it cost?

DataTalks.Club is the community behind the Zoomcamp series of courses. Alexey Grigorev, who founded the community, is one of the instructors. Videos, course materials and assignments are all free. If you run the code yourself, you may spend $1–5 on OpenAI credit.

The entry requirements are modest: you should write Python with confidence and have a laptop. Any laptop will do, and you do not need a GPU. For developers with a few years of backend experience, the technical bar is barely a hurdle. The harder part is the habit of measuring things, which the two later modules demand.

Two modules for week three

Module 4 (Evaluation) teaches how to measure retrieval quality and answer quality, both offline and online, for search, RAG and agents. Module 5 (Monitoring) turns to deployed systems. It covers user feedback, system health, real-time dashboards, metrics for each LLM call, latency and cost.

Read as a list, this sounds dry. It starts to mean something once you apply it to a specific incident.

Suppose you deploy an assistant that looks up internal procedures for a bank. In the first week, everyone is pleased with it. In the third week, a department head complains that its answers “sound plausible but are wrong”.

The usual reflex is to fix the prompt. But “wrong” has at least two possible sources. Either the retriever fetched the wrong documents, or it fetched the right ones and the model misread them. Changing the prompt only addresses the second.

Twenty questions to separate the two failures

This is the skill Module 4 builds, and you can practise it now. Write 20 questions like the ones the department head tends to ask, and for each one mark the passage of the procedure that holds the correct answer. That is your first offline evaluation set.

Step one: run only the retriever, without calling the model. For each question, check whether the correct passage appears in the results. Say 14 questions hit and 6 miss. Those 6 are retrieval failures. No prompt can rescue them, because the model never saw the right document.

Step two: look only at the 14 questions where the retriever succeeded. Give the model exactly that context and grade the answers. Any answer that is still wrong is an interpretation failure. Only now is it worth changing the prompt, changing how the context is presented or switching models.

After these two steps, the complaint that answers “sound plausible but are wrong” has become two separate numbers, each pointing to a layer that needs fixing. You go back to the department head with a diagnosis instead of a promise.

A hand-written test set goes stale

Twenty questions you wrote yourself will never match exactly what real users type. That is why Module 4 also covers online evaluation and Module 5 puts user feedback on the dashboard. A question that gets a low rating today should join the test set next week.

Monitoring also answers questions a test set leaves open: how often failures happen, and what each answer costs. For enterprise clients, an assistant that answers correctly but keeps users waiting, or runs up a large API bill, still has a problem.

Treat latency and cost as part of quality, on a par with accuracy.

Cohort or self-paced?

The course runs once a year and usually starts in June. “Live cohort” does not mean live classes. All lectures are pre-recorded, and the cohort adds deadlines and grading. Self-paced learners do not get a certificate, but they are free to build a portfolio project.

If you need deadlines to keep yourself going, wait for the next cohort. If you are already disciplined, start now from the GitHub repo. Either way, do not jump straight to Module 4. Move quickly through the basics, build a pipeline on a dataset you know well, then spend most of your time on evaluation and monitoring.

Turning the course into evidence on your CV

In an FDE interview, evidence that you have measured a real system counts for more than the name of a course. When you read job descriptions, look for phrases such as “evaluation”, “production monitoring” or “LLM observability”. That is exactly what Modules 4 and 5 cover.

On your CV, do not just write “completed LLM Zoomcamp”. Describe the RAG system you built, the evaluation set that separates retrieval from answers, and the dashboard tracking latency, cost and user feedback, with a link to the repo. In the interview, describe a time your test set showed you which layer a failure was in.

This course will not turn you into an FDE. But it teaches you to answer “how do you know?” with numbers, and for an FDE that is a large part of the job.