FDE Times dataFDE jobs open 881▼8 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 55Companies hiring 185Remote-friendly 25%Median US pay $201kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Tools

DSPy turns a client's eval set into a prompt optimiser

An algorithm once beat 'Let's think step by step'; DSPy applies that search to whole pipelines, so a model switch need not mean retuning by hand.

ShareLinkedInFacebookX

In brief

  • The APE paper found that LLM-generated instructions matched or beat those written by human annotators on 19 of 24 NLP tasks.
  • DSPy treats prompts as parameters: its compiler optimises the whole pipeline to maximise the metric you supply.
  • An optimiser is only as good as its metric and eval set, so the most valuable work an FDE does is defining 'correct' with the client.
Diagram of the DSPy loop in the article's hypothetical example: 300 labelled emails are split into 200 tuning emails and 100 holdout emails the optimizer never sees. The 200 emails go into the Optimizer. The Module (email to class) is compiled, the Optimizer tries instructions and few-shot examples, and the Metric (correct label = 1, wrong = 0) scores each candidate and returns the score. The FDE and the client must define what "correct" means themselves. The compiled prompt is scored on the 100 holdout emails: the hand-written prompt scores 78, the compiled one 86/100, or about 400 fewer misrouted emails a month at 5,000 emails a month.
With DSPy, the FDE's job is to define what "correct" means. The optimizer finds the wording, and the holdout set gives the final score. Figures are a hypothetical example from the article.

“Let’s think step by step”, the human-written chain-of-thought prompt, was once beaten by an algorithm using a language model. The Automatic Prompt Engineer (APE) method found a longer variant that asks the model to work through the problem step by step “to be sure we have the right answer”.

According to DAIR.AI’s Prompt Engineering Guide, this phrasing improved results on two problem sets, MultiArith and GSM8K.

The interesting part is not the wording but how it came about. Nobody sat down and thought of it. An algorithm searched for it and selected it, much as one tunes hyperparameters.

An FDE runs into this problem almost every week. At a client site, you can spend a whole week tuning prompts for a classification or extraction pipeline. Then the client switches models and everything has to be redone. DSPy turns that manual tuning into an optimisation problem, with the client’s eval set as the objective function.

A prompt is a parameter, not a work of craft

The idea of finding prompts algorithmically was set out clearly in “Large Language Models Are Human-Level Prompt Engineers” by Zhou et al., posted to arXiv on 3 November 2022. APE’s mechanism is simple: an LLM proposes a batch of candidate instructions, and the system searches that batch for the one with the highest score.

Across 24 NLP tasks, instructions generated by APE matched or beat those written by human annotators on 19.

DSPy applies the same logic at the level of the pipeline. The paper by Khattab, Zaharia, Potts et al., posted on 5 October 2023, describes a compiler that can optimise any DSPy pipeline to maximise a given metric.

DSPy modules are parameterised, meaning they can learn, for example by learning prompts or sets of few-shot examples.

The official Stanford NLP README describes DSPy as a framework for programming language models rather than prompting them. The full name is Declarative Self-improving Python, and according to the README it includes algorithms for optimising both prompts and model weights.

When should you reach for DSPy?

DSPy fits best when three conditions hold: the task has a measurable answer, the client has labelled data or is willing to produce it, and the pipeline will keep changing in model or requirements. Without the first, for instance when the client just wants answers that “sound better” and nobody can define what better means, the optimiser has nothing to hold on to.

Picture a bank that wants to sort complaint emails automatically into 6 categories so they reach the right department. The operations team labels 300 emails. You set aside 200 as optimisation data and keep the other 100 as a holdout that the optimiser never sees. The metric is a single function: 1 point if the predicted label matches the human label, 0 if not.

The sketch below shows those three pieces in code: a module that declares inputs and outputs, a metric function, and a compile call. It is illustrative only; check the optimiser class names and parameters against the documentation for the DSPy version you are using. (The identifiers and comments are in Vietnamese: classifier is the classifier, complaint_category the complaint category, correct_label the “correct label” metric, train_emails the 200 optimisation emails and optimized_program the optimised program.)

import dspy

# Module: declare the task, don't write a prompt
classifier = dspy.Predict("email -> complaint_category")

# Metric: 1 if it matches the human-assigned label, 0 otherwise
def correct_label(example, prediction, trace=None):
    return prediction.complaint_category == example.complaint_category

# Compile: the optimizer tries instructions and few-shot examples drawn from 200 emails
optimizer = dspy.BootstrapFewShot(metric=correct_label)
optimized_program = optimizer.compile(classifier, trainset=train_emails)

# Only report scores on the 100 holdout emails

Suppose the hand-written prompt scores 78/100 on the holdout. You run the compiler and let it try options, scoring each with the metric. If the best option reaches 86/100 on the holdout, the gap is 8 emails in every 100.

At 5,000 emails a month, the bank sends roughly 400 fewer emails to the wrong department each month, a number any head of operations grasps immediately.

Task Who does it
Define what “correct” means FDE with the client
Label data and split off the holdout Client, checked by the FDE
Generate and test prompt options Optimiser
Read the remaining errors on the holdout FDE

The last row is where the value lies. The 14 emails still misclassified usually point to an ambiguous category or inconsistent labelling, and that becomes the agenda for the next meeting with the client.

The optimiser is loyal only to the metric

DSPy optimises exactly what you measure, nothing more. If the metric checks only the output format and not the content, you will get a prompt that is very good at returning the right format. If the eval set is too small or skewed, the prompt will overfit to those few dozen examples.

That is why the holdout score, not the score on the optimisation data, is the number to report.

Cost also matters. Every option tried means another batch of model calls, so start with a small dataset and a cheap model before running at scale.

Even so, compared with the alternatives, prompt optimisation is the economical choice. The authors of GEPA, a reflective prompt-evolution method published in 2025 by Agrawal, Khattab et al., point out that reinforcement learning methods such as GRPO often need thousands of rollouts to learn a new task.

Across six tasks, GEPA beat GRPO by 6% on average and by up to 20% on some tasks, while using up to 35 times fewer rollouts. The lesson for FDEs: exhaust prompt optimisation before proposing fine-tuning to a client.

What to learn first to be useful straight away

Do not start with the API. Start with the skills of writing metrics and building eval sets, because those are the things DSPy will not do for you. Then read the APE abstract to grasp the propose-and-score loop, followed by the DSPy paper to see how modules, metrics and the compiler fit together.

When reading FDE or AI engineer job descriptions, look for phrases such as “eval”, “prompt optimization” or DSPy by name. On your CV, do not write “proficient in prompt engineering”. Write one sentence with numbers: how many examples your eval set had, that you used automatic optimisation, and how far holdout accuracy rose.

A hand-written prompt goes stale the moment the client changes models. A well-defined eval set stays useful, and next time you only need to rerun the compiler.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
5 sources
Read next on the roadmap · Stage 3: Applied AIWhat AutoGen's maintenance mode teaches FDEs about picking frameworksMicrosoft now steers new users to its Agent Framework, so ask first who will maintain a tool after you leave, then whether security will approve it.