# DSPy turns a client's eval set into a prompt optimiser

> An algorithm once beat 'Let's think step by step'; DSPy applies that search to whole pipelines, so a model switch need not mean retuning by hand.

Original: https://fdetimes.net/en/tools/dspy-automatic-prompt-optimization-evals/

"Let's think step by step", the human-written chain-of-thought prompt, was once beaten by an algorithm using a language model. The Automatic Prompt Engineer (APE) method found a longer variant that asks the model to work through the problem step by step "to be sure we have the right answer".

According to DAIR.AI's Prompt Engineering Guide, this phrasing improved results on two problem sets, MultiArith and GSM8K.

The interesting part is not the wording but how it came about. Nobody sat down and thought of it. An algorithm searched for it and selected it, much as one tunes hyperparameters.

An FDE runs into this problem almost every week. At a client site, you can spend a whole week tuning prompts for a classification or extraction pipeline. Then the client switches models and everything has to be redone. DSPy turns that manual tuning into an optimisation problem, with the client's eval set as the objective function.

## A prompt is a parameter, not a work of craft

The idea of finding prompts algorithmically was set out clearly in "Large Language Models Are Human-Level Prompt Engineers" by Zhou et al., posted to arXiv on 3 November 2022. APE's mechanism is simple: an LLM proposes a batch of candidate instructions, and the system searches that batch for the one with the highest score.

Across 24 NLP tasks, instructions generated by APE matched or beat those written by human annotators on 19.

DSPy applies the same logic at the level of the pipeline. The paper by Khattab, Zaharia, Potts et al., posted on 5 October 2023, describes a compiler that can optimise any DSPy pipeline to maximise a given metric.

DSPy modules are parameterised, meaning they can learn, for example by learning prompts or sets of few-shot examples.

The official Stanford NLP README describes DSPy as a framework for programming language models rather than prompting them. The full name is Declarative Self-improving Python, and according to the README it includes algorithms for optimising both prompts and model weights.

**Key point:** With DSPy, you do not hand the client a prompt. You hand over a metric, an eval set and a process that finds the prompt again whenever something changes.

## When should you reach for DSPy?

DSPy fits best when three conditions hold: the task has a measurable answer, the client has labelled data or is willing to produce it, and the pipeline will keep changing in model or requirements. Without the first, for instance when the client just wants answers that "sound better" and nobody can define what better means, the optimiser has nothing to hold on to.

Picture a bank that wants to sort complaint emails automatically into 6 categories so they reach the right department. The operations team labels 300 emails. You set aside 200 as optimisation data and keep the other 100 as a holdout that the optimiser never sees. The metric is a single function: 1 point if the predicted label matches the human label, 0 if not.

The sketch below shows those three pieces in code: a module that declares inputs and outputs, a metric function, and a compile call. It is illustrative only; check the optimiser class names and parameters against the documentation for the DSPy version you are using. (The identifiers and comments are in Vietnamese: `classifier` is the classifier, `complaint_category` the complaint category, `correct_label` the "correct label" metric, `train_emails` the 200 optimisation emails and `optimized_program` the optimised program.)

```python
import dspy

# Module: declare the task, don't write a prompt
classifier = dspy.Predict("email -> complaint_category")

# Metric: 1 if it matches the human-assigned label, 0 otherwise
def correct_label(example, prediction, trace=None):
    return prediction.complaint_category == example.complaint_category

# Compile: the optimizer tries instructions and few-shot examples drawn from 200 emails
optimizer = dspy.BootstrapFewShot(metric=correct_label)
optimized_program = optimizer.compile(classifier, trainset=train_emails)

# Only report scores on the 100 holdout emails
```

Suppose the hand-written prompt scores 78/100 on the holdout. You run the compiler and let it try options, scoring each with the metric. If the best option reaches 86/100 on the holdout, the gap is 8 emails in every 100.

At 5,000 emails a month, the bank sends roughly 400 fewer emails to the wrong department each month, a number any head of operations grasps immediately.

| Task | Who does it |
|---|---|
| Define what "correct" means | FDE with the client |
| Label data and split off the holdout | Client, checked by the FDE |
| Generate and test prompt options | Optimiser |
| Read the remaining errors on the holdout | FDE |

The last row is where the value lies. The 14 emails still misclassified usually point to an ambiguous category or inconsistent labelling, and that becomes the agenda for the next meeting with the client.

## The optimiser is loyal only to the metric

DSPy optimises exactly what you measure, nothing more. If the metric checks only the output format and not the content, you will get a prompt that is very good at returning the right format. If the eval set is too small or skewed, the prompt will overfit to those few dozen examples.

That is why the holdout score, not the score on the optimisation data, is the number to report.

Cost also matters. Every option tried means another batch of model calls, so start with a small dataset and a cheap model before running at scale.

Even so, compared with the alternatives, prompt optimisation is the economical choice. The authors of GEPA, a reflective prompt-evolution method published in 2025 by Agrawal, Khattab et al., point out that reinforcement learning methods such as GRPO often need thousands of rollouts to learn a new task.

Across six tasks, GEPA beat GRPO by 6% on average and by up to 20% on some tasks, while using up to 35 times fewer rollouts. The lesson for FDEs: exhaust prompt optimisation before proposing fine-tuning to a client.

## What to learn first to be useful straight away

Do not start with the API. Start with the skills of writing metrics and building eval sets, because those are the things DSPy will not do for you. Then read the APE abstract to grasp the propose-and-score loop, followed by the DSPy paper to see how modules, metrics and the compiler fit together.

When reading FDE or AI engineer job descriptions, look for phrases such as "eval", "prompt optimization" or DSPy by name. On your CV, do not write "proficient in prompt engineering". Write one sentence with numbers: how many examples your eval set had, that you used automatic optimisation, and how far holdout accuracy rose.

A hand-written prompt goes stale the moment the client changes models. A well-defined eval set stays useful, and next time you only need to rerun the compiler.

**Try this week:**

- Pick a prompt you are running in production, collect 50-100 labelled examples and write a metric function that returns correct/incorrect for each one.
- Split the data into an optimisation set and a holdout set, and record the hand-written prompt's holdout score as a baseline.
- Read the abstracts of the DSPy paper (arXiv 2310.03714) and APE (arXiv 2211.01910), then summarise each one's mechanism in three sentences.

## Sources

- [Automatic Prompt Engineer (APE) | Prompt Engineering Guide](https://www.promptingguide.ai/techniques/ape)

- [Large Language Models Are Human-Level Prompt Engineers](https://arxiv.org/abs/2211.01910)

- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://arxiv.org/abs/2310.03714)

- [GitHub - stanfordnlp/dspy: DSPy: The framework for programming—not prompting—language models](https://github.com/stanfordnlp/dspy)

- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://arxiv.org/abs/2507.19457)
