Distillation for FDEs: let a large model label the data, then put a small model in the customer's infrastructure

The demo runs on the largest model the customer trusts, and it performs well: it reads customer emails, assigns the right request type and explains its reasoning neatly. Then comes the meeting with the infrastructure team. They ask how much it will cost each month, how long each request will take, and why customer data has to leave the building.

The honest answer is usually that the large model is too expensive and too heavy for such a narrow task. Researchers from Google and Snorkel write that the computational demands of LLMs far exceed what most product teams can afford. At a customer with its own infrastructure and strict data rules, the problem is sharper still.

Distillation is how an FDE gets out of that bind. You keep the large model as the “teacher”, use it to generate data, then train a small model that runs on the customer’s machines. The skill sits between applied AI and deployment, and it decides whether a demo becomes a production system.

An idea that predates the LLM wave

In 2015 Geoffrey Hinton, Oriol Vinyals and Jeff Dean described exactly this problem with ensemble models. They acknowledged that making predictions with a whole ensemble is cumbersome, and showed that the knowledge in an ensemble can be compressed into a single model. That is the root of modern distillation.

The subtle point of the paper lies in “soft targets”: the student learns not only the correct label but the whole probability distribution the teacher produces. Imagine the teacher scores an email 0.7 as “claim request” and 0.25 as “fee complaint”. That 0.25 teaches the student that the two categories are close, something a hard label cannot convey.

With LLMs there is another signal: the explanation. The paper “Distilling Step-by-Step” (2023) uses LLM-generated rationales as additional supervision, training the small model in a multi-task setup: it predicts the label and generates the rationale.

Can a small model beat a large one?

On the paper’s benchmarks, yes. A fine-tuned 770-million-parameter T5 model outperformed few-shot-prompted PaLM, with 540 billion parameters, using only 80% of the available data. On average, their models achieved better results with less than 50% of the training examples.

Do not take that figure into a customer meeting as a promise. It is a benchmark, not your customer’s email. AWS also says student models on Bedrock respond much faster and cost less to run, but that is a vendor’s claim. The only thing you can promise is this: you will measure, on the customer’s own data.

That is why OpenAI’s official documentation on distillation sets a modest goal: train a small model to perform nearly as well as a larger, more expensive one on a specific task. And it lays down one principle: invest in fine-tuning only after you have evals.

A worked example: six email categories at an insurer

Suppose the customer is an insurance company that needs to sort emails into six groups, including “claim status enquiry” and “suspected fraud”. It has 20,000 unlabelled emails and wants the model to run on its internal cluster.

Step one is not calling the large model. You ask the customer’s specialists to label 600 emails and split them in two: 200 as a dev set for tuning the prompt, 400 as a holdout. The holdout is set aside, used neither for training nor for tuning the prompt.

If 60% of the holdout is “claim status enquiry”, a model that always answers with that label scores 60% accuracy without catching a single fraudulent email. So you must measure recall per class, especially for the class the customer most fears missing.

Step two: tune the teacher’s prompt and score it on the dev set until it reaches the eval threshold you and the customer agreed. This is exactly the workflow OpenAI recommends: get a prompt for the large model to pass the eval, collect its outputs, filter them into a dataset, then fine-tune a smaller model such as gpt-4.1-mini. If the teacher itself cannot reach the threshold, the student has nothing to learn.

Step three: have the teacher label all 20,000 emails with rationales, in a fixed format:

def label_with_teacher(email, teacher):
    out = teacher.generate(PROMPT, email)        # PROMPT already tuned on DEV_HUMAN
    rec = parse_json(out)                        # {"label": ..., "rationale": ...}
    if rec is None or rec["label"] not in LABELS:
        return None                              # drop malformed records
    return {"input": email, "label": rec["label"], "rationale": rec["rationale"]}

raw = [label_with_teacher(e, teacher) for e in emails]

Step four: filter the data, then fine-tune the student:

dataset = [r for r in raw if r]                                  # drop broken records
dataset = [r for r in dataset if agrees_on_rerun(r, teacher)]    # call again, keep consistent samples
train, val = split(dataset, 0.9)
student = finetune(base_small_model, train, targets=["label", "rationale"])

The two filtering lines are where many people cut corners. Malformed records, labels outside the list, and samples where the teacher answers differently on two calls are all noise. The student will diligently memorise that noise.

Step five: compare on the holdout before deploying. Only now are the 400 holdout emails opened, and you put three columns side by side: the teacher, the small model without fine-tuning, and the student.

OpenAI stresses that you need a reliable way to tell whether the fine-tuned model beats the base model, meaning the small model before fine-tuning. If the student does not clearly beat the untuned small model, the training cost was wasted.

Only if the student wins do you measure latency and resource use on the customer’s real hardware, then keep a small stream of traffic going to the teacher to detect when the data shifts.

When the infrastructure team asks about cost, put two figures side by side for the same volume of requests: the cost per request of continuing to call the teacher, and the cost per request of the student on the customer’s machines, plus the one-off cost of labelling and training.

Presented this way, the customer sees after how many requests distillation starts to pay off, rather than a vague promise that it will be “cheaper”.

Mistakes that make distillation fail silently

The worst mistake is scoring the student against the teacher’s labels. All you then measure is how well the student imitates its teacher, mistakes included. The holdout must be labelled by people who know the business.

A subtler variant is tuning the prompt on the holdout itself. After a few dozen rounds of edits, the prompt fits those exact 400 emails, and the final figure looks better than reality. The dev set exists so that the holdout can remain a genuine control.

The third mistake is forgetting that the labelling step also sends data somewhere. If emails may not leave the customer’s infrastructure, the teacher must run somewhere that has been approved, or you work only with an anonymised set.

Settle this with the security team before the first API call, and read the provider’s terms on using outputs to train other models carefully.

The last mistake is dropping the rationales to save tokens. The “Distilling Step-by-Step” results show that rationales are precisely what help a small model learn efficiently from less data. On the customer side, rationales also let specialists check why the model flagged an email.

Turning the skill into an edge in job applications

When reading job descriptions for FDE or applied AI roles, look for “fine-tuning”, “evals”, “on-prem” or “cost optimization” appearing together. That signals a company that needs someone who can run exactly this workflow. On a CV, a strong line states the task, the student’s size, its metric on a human-labelled holdout and the measured reduction in latency.

An exercise for this week: take a public classification dataset, pretend you have only 200 human labels, and split them into a dev set and a holdout.

Use a large model to label the rest of the data with rationales, fine-tune a small model, then write a one-page three-column comparison as above, with a paragraph on the class where the student performs worst.

The demo on a large model proves the problem can be solved. Distillation proves the customer can live with that solution every day, on its own machines.