W&B's eval course builds LLM judges, but judges still need humans
The Weave documentation adds what matters most at a client site: separating checks written in code from the judge, and treating the judge's prompt as code.
In brief
- Weights & Biases' LLM Apps: Evaluation course teaches how to build, optimize and scale LLM judges.
- The Weave documentation adds the part worth keeping: let code score first, the judge second, and manage the judge's prompt like code.
- Treat with caution the promise of minimal human effort, because the client still decides what good looks like.
Weights & Biases pitches its “LLM Apps: Evaluation” course with an ambitious goal. According to the announcement, last updated on January 8, 2025, the course covers techniques for building, optimizing and scaling AI evaluators with minimal human effort. For a forward deployed engineer, that sentence is both an invitation and a trap.
The invitation is that almost every client asks the same question after a demo: how do we know this system is good enough for production? The trap is the “minimal human” part. At a client site, the person who can define what “good” means is usually one of their domain experts, not another model.
So the course is best taken alongside the Weave documentation. The course concentrates on LLM judges; the Weave docs show how to combine code-based checks with a judge into a hybrid evaluation pipeline. The promise of automation deserves some distance.
What skill does the course actually teach?
W&B presents this as a course on building LLM judges. The official course page on wandb.ai returns a 404 error, so for now only the company’s announcement confirms the course’s name and content. Chapter 3, led by Ayush Thakur, focuses on how to structure an LLM evaluator.
According to W&B, the chapter’s approach turns subjective criteria such as grammar, coherence and logic into checks that are repeatable and scalable.
The tooling underneath is W&B’s Weave. The company’s evaluation product page defines evaluation as measuring how well AI agents and applications perform across multiple dimensions, using benchmarks, real-world scenarios and comparative analysis. The central unit is the scorer, which takes an application’s inputs and outputs and returns metrics.
The FDE skill the course trains is turning a vague client requirement, such as “answers must sound professional”, into a measurement that can be rerun every time the prompt or the model changes.
Two kinds of scorer, two kinds of question
The Weave documentation splits scorers into two camps. On the code side is the function-based scorer: an ordinary Python function decorated with @weave.op that returns a dictionary. The example in the docs is simple enough to be easily dismissed: checking whether a piece of text is written entirely in capitals.
On the other side is LLM-as-a-judge. The scoring prompt can reference variables from the op being evaluated. For this kind, the docs recommend a class-based Scorer, partly because it makes it easier to try different prompts for the judge.
Code-based scorer
- Python function decorated with @weave.op
- Cheap, fast, same input always gives the same result
- Suits formats, required fields, length
LLM judge
- Class-based Scorer with a scoring prompt
- Costs model calls, results can vary
- Suits coherence, logic, tone
Take a hypothetical chatbot that summarizes claim files for an insurance company, with a set of 200 test cases.
The code-based scorer checks whether the output parses as JSON, whether it contains a correctly formatted claim ID and whether it exceeds the length limit.
If 30 cases fail at this stage, the fault is already clear, and there is no need to spend judge calls on them.
Only the remaining 170 cases go to the judge, to be scored for coherence and logical correctness. A judge should answer only the questions code cannot.
Three ideas to take to a client site
The first is the ordering in the example above: code first, judge second. Cheap and certain comes before expensive and variable. When you report to the client, you can then separate formatting errors, which can be fixed in an afternoon, from quality problems, which need a discussion with them.
The second is to break subjective criteria apart.
Chapter 3 shows that grammar, coherence and logic can each become repeatable checks. The next step is to give each criterion its own scorer, rather than asking a judge something generic such as “is this answer good?” and getting back a single overall score.
When a score drops, you know at once which dimension broke.
The third: the judge’s prompt is code too. The advice to use a class-based Scorer for trying several prompts implies that a judge needs versioning, comparison and iteration like every other component of the system.
What to add yourself: human raters
The goal of “minimal human effort” makes sense at the scaling stage, once a judge is running reliably across thousands of outputs. At the definition stage it does not.
Before using a judge’s scores in a client report, ask their experts to label a small sample and compare those labels with the judge’s scores. If the two disagree, adjust the prompt before scaling up.
This is also the step that earns the client’s trust. A number their own team has confirmed is more persuasive than any dashboard chart.
Who should take it, and in what order
The course suits engineers who already have an LLM application running but no evaluation suite.
Start with the scorers overview in the Weave documentation (docs.coreweave.com/weave/guides/evaluation/scorers) and write a function-based scorer for your own project.
Then study Chapter 3 on structuring an evaluator, and only after that move on to class-based judges.
On the career side, when reading FDE job postings, look for phrases such as “evals” or “LLM-as-a-judge”. On a CV, avoid the generic “experience evaluating LLMs”. Describe the pipeline you built instead: how many code-based scorers, which criteria the judge scored, and how you checked the judge against human labels.
The course teaches you to build a judge. At a client site, the harder job is proving that the judge scores correctly, and that is where the FDE work begins.
Was this article useful?
Thanks for the feedback!