Rob Zuber says LLM testing already has 90% of the tools it needs

Rob Zuber, CTO of CircleCI, makes a bold claim about testing LLM applications: in his view, we already have 90% of the tools and methods we need. To him, CI/CD is a natural fit for testing LLM-powered applications. It is an evolution of an old craft, not a new discipline.

That is also the argument of Automated Testing for LLMOps, a short course taught by Zuber, built by DeepLearning.AI with CircleCI and released in January 2024. It is a beginner-level course and runs to about an hour. Its main goal is to help you build a CI workflow that evaluates an LLM application.

For an FDE, that hour is well spent. Once a client’s agent or RAG system is in production, the question you will hear most often is “how do you know this prompt change didn’t break something else?”. The course gives you a way to answer with a pipeline rather than a promise.

What does the course teach in an hour?

The syllabus is compact: an introduction, an overview of CI, an overview of automated evals, model-graded evals, a comprehensive testing framework, a conclusion, a quiz and an optional walkthrough of the CircleCI configuration file. Anyone can enrol, but the graded exercises require a PRO account.

It is aimed at people with basic Python who have already built an LLM-based application. If you have never called a model’s API, build a small application first: the course assumes you already have something to test.

The content centres on familiar LLM failures: hallucination, data drift and harmful or offensive output. The course starts from a simple observation: LLMs behave less predictably than traditional software.

First idea: two kinds of eval, two cadences

The idea most worth keeping is how the course splits evals into two kinds. Rule-based evals match strings or patterns, for example with regular expressions, and run very fast. Model-graded evals are for problems where many good or bad outputs are equally valid, and the course uses them to detect hallucination.

Imagine you are deploying a bot that answers questions about refund policy for a retailer. Rule-based evals check what a machine can measure: the output is valid JSON, order IDs are in the right format, internal email addresses never leak.

Model-graded evals answer the question a regex cannot: has the bot invented a “refunds within 60 days” term that appears nowhere in the policy documents?

A practical arrangement follows. The cheap, fast kind runs on every commit; the expensive, slow kind runs at more important points, such as before a merge or a release. This is operational advice rather than something the course prescribes, but it follows directly from the nature of the two kinds of eval the course describes.

Second idea: evals are tests, so they belong in CI

The second idea lies in Zuber’s framing. If evals are just a notebook someone opens now and then, they will be forgotten after the first week at the client. Putting evals into CI turns answer quality into a gate, like unit tests: fail, and you do not merge.

This is a language the client’s engineering team understands immediately. You do not need to persuade them to adopt an unfamiliar “LLMOps” process. You simply add a few jobs to the pipeline they already run.

The third idea, and what to leave behind

The third idea the course only touches on; CircleCI’s own production experience spells it out more clearly. According to people at CircleCI, non-determinism is the main challenge, and no single eval is perfect, so several kinds must be combined.

They also recommend building tools into the application to capture user feedback, then feeding that feedback back into the eval suite.

A course about CI alone will not teach that loop, yet it is exactly the part FDEs must do in the field. Cases where the bot answered wrongly and a real user clicked 👎 should become new test cases in next week’s eval suite.

What to leave behind is the tooling. Both kinds of eval in the course run automatically on CircleCI, while your client may use something else.

So keep the thinking and rewrite the configuration yourself. A concrete exercise with GitHub Actions: create a first job that runs the rule-based evals on every push, and a second job that runs only on pull requests to the main branch, declares a dependency on the first job and calls the grading model on a fixed set of questions.

Finally, turn on branch protection so both jobs must be green before anything can merge.

Who should take it, and in what order?

If you are a software engineer who has written CI but never evaluated an LLM, this is the right way in: take this course first, then look for deeper material on LLM-as-judge. If you are already an FDE running evals by hand, skip the CI introduction and go straight to model-graded evals and the comprehensive testing framework.

On your CV, do not write “completed Automated Testing for LLMOps”. Write a sentence with an outcome, such as “built a CI pipeline running rule-based and model-graded evals for a RAG application, blocking merges when the hallucination rate exceeds a threshold”, with a link to the repo.

When an FDE job posting mentions “evals” or “LLM testing” alongside “CI/CD”, have such a pipeline ready to describe in detail if asked.

Zuber may be right about the 90%. But at the client, the missing 10% often decides whether you are given more work.