Chip Huyen's CS 329S: the Stanford course on evaluating and monitoring ML systems in production

At Stanford’s CS 329S, 65% of the grade does not come from exams. It comes from a final project: building a real ML application and presenting it at a demo day. The grading scheme says a lot about what the course values.

The course’s full name is Machine Learning Systems Design, and it is taught by Chip Huyen. The course website notes that this is the second time it has been offered.

A schedule running from 3 January to 9 March, and a 2022 email address for the teaching staff, indicate this is the winter 2022 edition. The stated goal is clear: to provide an iterative framework for developing ML systems that run in the real world.

If you are an FDE, or about to become one, “iterative” is the word that matters most. The reason is in the course description itself: data distributions shift constantly, so a model that performs well at handover may not stay good once the customer’s data changes.

The course teaches the work that comes after handover

The CS 329S syllabus devotes separate sessions to offline evaluation and model evaluation. The model evaluation session includes a tutorial on RecList. That is followed by a session on diagnosing ML system failures, distribution shift and monitoring, and then a monitoring tutorial using WhyLogs and Evidently. Production monitoring gets a full session, rather than a few notes at the end of a chapter.

The syllabus also has a session called “ML beyond accuracy”, covering fairness, security and governance. The closing session deals with the business value of AI, the part closest to what an FDE has to present to customers. The sequence mirrors a real engagement: make it measurable, make it operable, and only then prove its value.

An ML system is a loop

The course website says it teaches how to continually monitor and deploy changes to ML systems, because data distributions do not stand still. The official repository for the companion book describes the aim as systems that are reliable, scalable, maintainable and able to adapt as the environment and business requirements change.

Chip Huyen’s page for the book is more specific still: how you choose training data, how often you retrain and what you monitor all affect the entire system.

For an FDE, these are questions to answer in the deployment plan, not after the first incident. When you write the scope with a customer, include a separate section on retraining frequency and who is responsible for monitoring.

Measure offline so the customer does not have to do it for you

The course gives offline evaluation its own session, and the reason is easy to see: a single number on a test set is rarely enough to make a customer trust a model. Imagine you are about to hand over a product recommendation model to a retail chain. The overall test score looks fine.

Before handover, you split the test set by customer segment and score it again. The results may reveal that new customers, precisely the segment the marketing team cares about most, get markedly worse recommendations. Finding that at the offline stage is far cheaper than letting the customer discover it after the model is live.

The RecList tutorial in the model evaluation session is the place to build this habit of slice-based evaluation. The “ML beyond accuracy” session covers the rest: the questions on fairness, security and governance that the customer’s legal team will inevitably ask.

Detecting change is easy; choosing which changes to act on is hard

This is the idea most worth keeping. In a post on real-time machine learning published on 2 January 2022, Chip Huyen wrote that the problem is not detecting changes but knowing which changes actually need your attention. The same post ties retraining to detected distribution shift.

Imagine Evidently flags drift on dozens of features after one of the customer’s promotions. If you retrain on every alert, the customer’s team will soon ignore all of them. A better approach is to agree with the customer from the start: changes that affect business metrics trigger retraining; other changes are simply logged.

In what order should you study it?

The course lectures were expanded into the book Designing Machine Learning Systems (O’Reilly, 2022). So read the book first to get the full picture. Then open the syllabus at stanford-cs329s.github.io, work through the RecList tutorial from the model evaluation session, and only then move on to the monitoring tutorial with WhyLogs and Evidently.

The final step is a project of your own modelled on the course’s final project, the part worth 65% of the grade. Build a small ML application, deploy it, add monitoring, then demo it to someone else. If the job you are aiming for involves putting models onto customers’ real data, this course is worth your time.

Two mistakes are easy to make when working on your own. The first is reporting only a single overall number, exactly the trap the retail example above illustrates. The second is wiring drift alerts directly to a retraining command, so the system retrains constantly and nobody knows why.

When you put the project on your CV, do not just write “built a model achieving X%”. Write which slices you evaluated the model on, which tool you used to monitor drift, and what rule decides when to retrain. Those lines show you have operated a system, not merely trained a model.

A highly accurate model proves only its first day. The CS 329S syllabus devotes entire sessions to what happens next.