# Catch prompt regressions in the pull request, before the client does

> A GitHub Actions workflow runs promptfoo to compare old and new prompts on the same golden dataset, then posts the scores as a comment on every PR that edits a prompt.

Original: https://fdetimes.net/en/guides/promptfoo-regression-evals-github-actions/

Take a team that edits one sentence in a system prompt so the chatbot gives shorter answers. The demo looks fine. Two weeks later, the client reports that the bot has stopped mentioning the order number in its replies, something their operations team relies on every day.

Traditional unit tests do not catch this kind of failure. An ordinary software function has one correct output for each input; with an LLM, as Evidently AI points out, the same input can produce many acceptable answers, so exact matching is useless.

Shipping to production does not end the testing either: offline evals are still needed to run regression tests after every change. For an FDE, who changes prompts and models directly on a client's system, that safety net is mandatory.

This guide builds that net: a GitHub Actions workflow that runs promptfoo on every pull request that edits a prompt.

## What you will build, and what you need

The end result: whenever someone opens a PR that changes something in the `prompts/` directory, GitHub Actions runs an eval comparing the old prompt with the new one on the same golden dataset, then comments the results directly on the PR. If the score falls below a threshold, you get a red signal before merging.

You need a GitHub repo where you have admin rights, an API key from a model provider (the example below uses OpenAI), and a directory holding your prompts. You should be able to read YAML and understand how pull requests work.

## Step 1: the golden dataset matters more than the YAML

A set of reference inputs with approved outputs is usually called a "golden dataset", and the whole test suite revolves around it. So do not rush into writing the workflow. Start with 15-20 real questions, taken from logs or from meetings with the client, each paired with the traits a correct answer must have.

Take a delivery company. The question "Where is my order DH-1042?" has no single right answer. But every good answer must repeat the order number, must not invent a delivery date, and must stay polite. Those three traits become three assertions.

Check after this step: every test case has at least one condition a machine can verify (it contains a certain string) and one that needs judgment (tone, factual accuracy).

## Step 2: write the promptfoo config with weighted assertions

In promptfoo, each assertion compares the LLM's output with an expected value or condition; pass or fail is derived from a weighted score measured against a threshold. That threshold gives you the pass/fail signal in CI. The file below is trimmed to illustrate the structure; check the field names against promptfoo's Assertions and Metrics page before using it.

```yaml
# prompts/promptfooconfig.yaml (trimmed)
prompts:
  - file://support_prompt.txt
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      question: "Where is my order DH-1042?"
    threshold: 0.75
    assert:
      - type: contains
        value: "DH-1042"
        weight: 1
      - type: llm-rubric
        value: "Does not invent a specific delivery date"
        weight: 2
      - type: llm-rubric
        value: "Polite, concise tone"
        weight: 1
```

A quick calculation shows how the threshold works, assuming the score is computed as a weighted average. The total weight is 4. The old prompt covers all three points: 4/4 = 1.0, pass.

Now suppose the new prompt, pushed to answer briefly, makes up a delivery date. The output still has the order number (1) and is still polite (1), but it loses the 2 points of the heaviest assertion: 2/4 = 0.5, below the 0.75 threshold, and the test fails. If the new prompt only lost its politeness, the score would be 3/4 = 0.75, exactly at the threshold.

| Assertion (weight) | Old prompt | New prompt that invents a delivery date |
|---|---|---|
| Contains "DH-1042" (1) | Pass: 1 | Pass: 1 |
| Does not invent a delivery date (2) | Pass: 2 | Fail: 0 |
| Polite, concise (1) | Pass: 1 | Pass: 1 |
| Score against 0.75 threshold | 4/4 = 1.0, pass | 2/4 = 0.5, fail |

The table shows why a change that looks harmless in a demo still gets flagged red: a single serious error is enough to pull the score below the threshold.

**Key point:** Weights are where you encode the client's priorities: making up information must cost more than a lapse in style.

Check: run a case you know is wrong and confirm it fails. A test suite that never goes red is testing nothing.

## Step 3: a workflow that runs only when prompts change

Every eval run means billable API calls, so you do not want it firing when someone edits the README. The `paths` filter on the `pull_request` event handles this: the workflow runs only when at least one changed file matches the pattern.

```yaml
# .github/workflows/prompt-eval.yml
name: Prompt regression eval
on:
  pull_request:
    paths:
      - 'prompts/**'
concurrency:
  group: prompt-eval-${{ github.ref }}
  cancel-in-progress: true
permissions:
  contents: read
  pull-requests: write
```

The `concurrency` block deals with another source of cost. Within a single concurrency group, at most one job or workflow runs at any moment. When you push three commits in a row to the same PR, the older run is canceled instead of all three calling the API.

The `pull-requests: write` permission is essential: promptfoo needs it to post the comment on the PR.

## Step 4: add the promptfoo action and the secret

The promptfoo action does the heavy lifting: on every PR that edits a prompt, it runs a full before-and-after comparison and posts the results to the PR. If you use OpenAI, set the `OPENAI_API_KEY` secret in the repository (Settings, under Secrets); for other providers, use the matching key.

```yaml
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: promptfoo/promptfoo-action@v1
        with:
          # key/token input names: check the promptfoo-action README
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          prompts: 'prompts/**/*.txt'
          config: 'prompts/promptfooconfig.yaml'
```

The two inputs `prompts` (a glob) and `config` (the path to the config file) are the core of promptfoo's example. The checkout step and the way the key is passed are simplified here; read the Testing Prompts with GitHub Actions page for the full configuration for the version you use.

Check: open a PR that changes a single word in `support_prompt.txt`. The Actions tab should show the workflow running, and the PR should get a comment with a comparison table. Then open a PR that only edits the README: the workflow must not run.

A note on scope: this guide stops at a red signal on the PR, which is not yet a merge block. Before promising a client that a bad prompt "cannot reach main", verify in your own repo whether the job reports a failure when the score falls below the threshold.

## Mistakes that make the pipeline run without meaning anything

The most common mistake is a golden dataset that is too easy. If all 20 questions are sample questions from the demo, the eval will stay green forever while real users ask about other things. Add every question that has ever caused an incident.

The second: the glob in `paths` does not match where the prompts actually live. Many teams keep prompts inside Python files, edit them, and the workflow stays silent. Test it by editing that exact file and seeing whether Actions runs.

The third is forgetting that changing the model is also a change. If the model name sits in `promptfooconfig.yaml` inside the `prompts/` directory, a model switch triggers the eval automatically. If it lives in a different config file, add that path to `paths`.

## What this skill looks like at a client site

The purpose of regression testing is to prove that a new change improves the system without breaking what used to work, and the most sensible place for it is CI/CD, running after every change.

Its value becomes obvious when, say, the client wants to switch to a cheaper model: instead of arguing on gut feeling, you open a PR that changes the model and show them the comparison comment.

The first thing to do with a new client: sit down with the operators, ask for the 20 questions that worry them most, and ask which errors are unacceptable. Their answer sets the weights. That list of questions, together with its weights, is the most valuable thing to hand over when you leave the project.

## How to put it on your CV and take it into interviews

When reading FDE job descriptions, watch for phrases such as "evaluation", "LLM testing" and "CI/CD". Then compare two ways of describing the same experience on a CV.

The second version is more convincing because a recruiter can check it. Include a link to a public PR with the results comment, ideally one flagged red because the new prompt was worse.

In interviews, be ready to answer "When you change models, how do you know nothing breaks?" with the DH-1042 example and the 2/4 calculation above.

And turn the question back on the employer: "How does the team regression-test prompts before deploying to clients?" The answer tells you whether you will get to build this safety net, or spend your time cleaning up after its absence.

Prompts will change dozens of times after you leave a project. What you leave the client is not the best prompt, but the test every future prompt must pass to prove it has not broken what already worked.

**Try this week:**

- Pick 15-20 real questions from an LLM project you are working on, record the approved outputs and save them as a golden dataset in the repo.
- Add a promptfoo workflow with a paths filter pointing at the prompts directory, open a PR that edits a prompt and read the results comment.
- Deliberately write a worse prompt, confirm the PR is flagged red, then save the link to that PR as evidence for your portfolio.

## Sources

- [LLM evaluation: a beginner's guide (Evidently AI)](https://www.evidentlyai.com/llm-guide/llm-evaluation)

- [Watch the language: A tutorial on regression testing for LLMs](https://www.evidentlyai.com/blog/llm-regression-testing-tutorial)

- [Workflow syntax for GitHub Actions](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions)

- [Testing Prompts with GitHub Actions | Promptfoo](https://www.promptfoo.dev/docs/integrations/github-action/)

- [Assertions and Metrics - LLM Output Validation | Promptfoo](https://www.promptfoo.dev/docs/configuration/expected-outputs/)
