Aurélien Géron's Hands-On Machine Learning: FDEs should start at chapter 2

The book has 19 notebooks in all, and the second carries the driest name: end_to_end_machine_learning_project. Alongside all the chapter’s example code and exercise solutions, it contains a section called “Create a Test Set” and a line that sets a random seed so that every rerun produces exactly the same result.

For a Forward Deployed Engineer, this is the chapter to read first. At a client, the first week rarely begins with choosing a model architecture. It begins with a data file and an evaluation number that sooner or later you will have to defend.

Chapter 2 teaches you how to make that number hold up.

Which book, which repo?

The book is Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow by Aurélien Géron, published by O’Reilly and now in its third edition. The resource guide of the Washington and Lee University library also lists it among recommended machine learning books.

The best way to study it is through the official ageron/handson-ml3 repo on GitHub. According to the author, the repo holds the example code and exercise solutions for the third edition, and its aim is to teach the fundamentals of machine learning in Python.

The chapter 2 notebook contains all the chapter’s example code and solutions. So you learn by running it cell by cell, not by reading it cover to cover like a novel.

Why is the test set an FDE’s problem?

The official scikit-learn documentation describes holding out part of the data as a test set as common practice in supervised learning experiments. It comes with a very short rule: never use test data to make decisions about the model, and never call fit on test data.

That sounds obvious. But the moment a client is pressing for results is exactly when the rule is easiest to skip.

scikit-learn calls the consequence data leakage: the model is built using information that will not be available when it makes real predictions. The evaluation number looks excellent; the model in production does not.

A delivery company with 50,000 orders

Imagine a delivery company hands you 50,000 orders and asks you to predict which ones will be late. The leaky approach is to inspect all 50,000 rows, pick features based on what you just saw, normalise across the whole dataset, and only then make an 80/20 split.

Both mistakes break scikit-learn’s rule. Choosing features after looking at the 10,000 test rows is using test data to make a model decision; normalising across the whole dataset is calling fit on test data.

It follows from that rule that the safe approach is to split before preprocessing, and ideally before you even start exploring the data to choose features.

Skip this and you present a fine-looking number, then the following week the model performs much worse on new orders. The person who championed the project on the client side will be the first to lose face.

The clean approach is to split at the very start, lock away the 10,000 test rows, and fit preprocessing steps only on the 40,000 training rows. Set a random seed, as the notebook does, so a colleague who reruns it gets exactly the same split.

The chapter 2 notebook creates its test set with a stratified split and notes that doing a single such split with scikit-learn is far more concise. Suppose 5% of orders are late, or 2,500 orders. A stratified split keeps that 5% ratio on both sides, so the test set will contain about 500 late orders rather than being skewed by chance.

But orders have dates. scikit-learn warns that shuffling time-ordered data can easily produce an overfit model and inflated validation scores. The TimeSeriesSplit tool does the opposite: it trains on the past, tests on the future, and the training set grows over time.

For the delivery company, using the most recent month as the test set is usually more honest. Settle this with the client, because only they know when and for what the model will be used.

Who should read it, and in what order?

If you are a software engineer moving into FDE work with no ML experience, start with chapter 2. Run the notebook, do the exercises yourself and only then open the solutions. The Keras and TensorFlow sections can wait until a real client needs them.

If you are already an ML engineer, skim chapter 2 as a checklist. The question is not whether you know the steps. It is whether you still carry them all out when a client is pushing you.

Which three ideas are worth keeping?

The first is that the split must be reproducible: with a seed, anyone who reruns it gets the same number. The second is to split before preprocessing, let test data drive no decision, and never fit on test.

The last is that the split must suit the data: stratified when there are rare groups, time-based when the model predicts the future.

To test yourself, before running the cell that creates the test set, write one sentence explaining why you are splitting that way. If the sentence still holds when you swap the book’s data for the 50,000 orders above, you have understood chapter 2.

If you are preparing to apply for FDE roles, redo chapter 2 on a public dataset from the industry you are targeting and put it on GitHub. When a job description mentions “end-to-end” or “model evaluation”, a README that states clearly how you split the test set and why is more concrete evidence than any line of self-description.

Neural networks are what catch a client’s attention. What keeps their trust is a test set you have never touched.