DSPy turns a client's eval set into a prompt optimiser
An algorithm once beat 'Let's think step by step'; DSPy applies that search to whole pipelines, so a model switch need not mean retuning by hand.
In brief
- The APE paper found that LLM-generated instructions matched or beat those written by human annotators on 19 of 24 NLP tasks.
- DSPy treats prompts as parameters: its compiler optimises the whole pipeline to maximise the metric you supply.
- An optimiser is only as good as its metric and eval set, so the most valuable work an FDE does is defining 'correct' with the client.
“Let’s think step by step”, the human-written chain-of-thought prompt, was once beaten by an algorithm using a language model. The Automatic Prompt Engineer (APE) method found a longer variant that asks the model to work through the problem step by step “to be sure we have the right answer”.
According to DAIR.AI’s Prompt Engineering Guide, this phrasing improved results on two problem sets, MultiArith and GSM8K.
The interesting part is not the wording but how it came about. Nobody sat down and thought of it. An algorithm searched for it and selected it, much as one tunes hyperparameters.
An FDE runs into this problem almost every week. At a client site, you can spend a whole week tuning prompts for a classification or extraction pipeline. Then the client switches models and everything has to be redone. DSPy turns that manual tuning into an optimisation problem, with the client’s eval set as the objective function.
A prompt is a parameter, not a work of craft
The idea of finding prompts algorithmically was set out clearly in “Large Language Models Are Human-Level Prompt Engineers” by Zhou et al., posted to arXiv on 3 November 2022. APE’s mechanism is simple: an LLM proposes a batch of candidate instructions, and the system searches that batch for the one with the highest score.
Across 24 NLP tasks, instructions generated by APE matched or beat those written by human annotators on 19.
DSPy applies the same logic at the level of the pipeline. The paper by Khattab, Zaharia, Potts et al., posted on 5 October 2023, describes a compiler that can optimise any DSPy pipeline to maximise a given metric.
DSPy modules are parameterised, meaning they can learn, for example by learning prompts or sets of few-shot examples.
The official Stanford NLP README describes DSPy as a framework for programming language models rather than prompting them. The full name is Declarative Self-improving Python, and according to the README it includes algorithms for optimising both prompts and model weights.
When should you reach for DSPy?
DSPy fits best when three conditions hold: the task has a measurable answer, the client has labelled data or is willing to produce it, and the pipeline will keep changing in model or requirements. Without the first, for instance when the client just wants answers that “sound better” and nobody can define what better means, the optimiser has nothing to hold on to.
Picture a bank that wants to sort complaint emails automatically into 6 categories so they reach the right department. The operations team labels 300 emails. You set aside 200 as optimisation data and keep the other 100 as a holdout that the optimiser never sees. The metric is a single function: 1 point if the predicted label matches the human label, 0 if not.
The sketch below shows those three pieces in code: a module that declares inputs and outputs, a metric function, and a compile call. It is illustrative only; check the optimiser class names and parameters against the documentation for the DSPy version you are using. (The identifiers and comments are in Vietnamese: classifier is the classifier, complaint_category the complaint category, correct_label the “correct label” metric, train_emails the 200 optimisation emails and optimized_program the optimised program.)
import dspy
# Module: declare the task, don't write a prompt
classifier = dspy.Predict("email -> complaint_category")
# Metric: 1 if it matches the human-assigned label, 0 otherwise
def correct_label(example, prediction, trace=None):
return prediction.complaint_category == example.complaint_category
# Compile: the optimizer tries instructions and few-shot examples drawn from 200 emails
optimizer = dspy.BootstrapFewShot(metric=correct_label)
optimized_program = optimizer.compile(classifier, trainset=train_emails)
# Only report scores on the 100 holdout emails
Suppose the hand-written prompt scores 78/100 on the holdout. You run the compiler and let it try options, scoring each with the metric. If the best option reaches 86/100 on the holdout, the gap is 8 emails in every 100.
At 5,000 emails a month, the bank sends roughly 400 fewer emails to the wrong department each month, a number any head of operations grasps immediately.
| Task | Who does it |
|---|---|
| Define what “correct” means | FDE with the client |
| Label data and split off the holdout | Client, checked by the FDE |
| Generate and test prompt options | Optimiser |
| Read the remaining errors on the holdout | FDE |
The last row is where the value lies. The 14 emails still misclassified usually point to an ambiguous category or inconsistent labelling, and that becomes the agenda for the next meeting with the client.
The optimiser is loyal only to the metric
DSPy optimises exactly what you measure, nothing more. If the metric checks only the output format and not the content, you will get a prompt that is very good at returning the right format. If the eval set is too small or skewed, the prompt will overfit to those few dozen examples.
That is why the holdout score, not the score on the optimisation data, is the number to report.
Cost also matters. Every option tried means another batch of model calls, so start with a small dataset and a cheap model before running at scale.
Even so, compared with the alternatives, prompt optimisation is the economical choice. The authors of GEPA, a reflective prompt-evolution method published in 2025 by Agrawal, Khattab et al., point out that reinforcement learning methods such as GRPO often need thousands of rollouts to learn a new task.
Across six tasks, GEPA beat GRPO by 6% on average and by up to 20% on some tasks, while using up to 35 times fewer rollouts. The lesson for FDEs: exhaust prompt optimisation before proposing fine-tuning to a client.
What to learn first to be useful straight away
Do not start with the API. Start with the skills of writing metrics and building eval sets, because those are the things DSPy will not do for you. Then read the APE abstract to grasp the propose-and-score loop, followed by the DSPy paper to see how modules, metrics and the compiler fit together.
When reading FDE or AI engineer job descriptions, look for phrases such as “eval”, “prompt optimization” or DSPy by name. On your CV, do not write “proficient in prompt engineering”. Write one sentence with numbers: how many examples your eval set had, that you used automatic optimisation, and how far holdout accuracy rose.
A hand-written prompt goes stale the moment the client changes models. A well-defined eval set stays useful, and next time you only need to rerun the compiler.
Was this article useful?
Thanks for the feedback!
5 sources
- Automatic Prompt Engineer (APE) | Prompt Engineering Guide
- Large Language Models Are Human-Level Prompt Engineers · 2022-11-03
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines · 2023-10-05
- GitHub - stanfordnlp/dspy: DSPy: The framework for programming—not prompting—language models
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning · 2025-07-25