Turning a client's SOP into a prompt: write the tests first, the prompt second

OpenAI’s API documentation contains a sentence worth pinning to every FDE’s monitor: LLM output is non-deterministic, and model behaviour changes between snapshots and model families. A prompt that runs smoothly in today’s client demo can drift next month without anyone touching a word.

When working with a client, the best source of instructions is often already in their hands: the standard operating procedure, or SOP. New staff read it to learn how to handle a refund request or classify a complaint. Your job is to turn that same document into a prompt the model follows consistently, and to prove that it does.

This article walks through a hypothetical example from start to finish. The overall framework comes from the five steps in Google’s Prompting Essentials course (task, context, references, evaluate, iterate). The detail comes from Anthropic’s prompt-writing guidance.

What you will build, and what you need

Imagine the client is an e-commerce marketplace with an SOP for handling refund requests. You need to produce two things. The first is a prompt that takes a buyer’s request and returns a decision — approve, reject or escalate to a human — with a reason. The second is a set of test cases to grade that prompt.

You need the latest version of the SOP and a batch of past cases that staff have handled, with the correct decision recorded. You also need access to a model, and a colleague who has never read this SOP. Do not skip the colleague: you will need them in step 6.

Step 1: Write the test set before the prompt

The natural reflex is to open an editor and start writing the prompt. Do the opposite. OpenAI advises starting with evals that measure model output to establish a baseline, then iterating: revise the prompt, measure again. With an SOP, you build the evals from the client’s historical cases; the baseline number arrives in step 2, when you run the first prompt through this test set.

The format below is simply one way to organise the cases for illustration, not a required standard. Each line holds a case ID, the input and the expected decision (approve = approve, reject = reject, escalate = escalate to a human). The inputs read, in turn: an order delivered six days late, buyer asks for a refund; the buyer has used the product and wants a refund after changing their mind; a high-value order where the photo proving the defect is blurry.

{"id": "c01", "input": "Order delivered 6 days late, buyer requests a refund", "expected": "approve"}
{"id": "c02", "input": "Buyer has already used the product, requests a refund after changing their mind", "expected": "reject"}
{"id": "c03", "input": "High-value order, photo proving the defect is blurry", "expected": "escalate"}

Check: every case must have an answer confirmed by the client’s operations lead, not one you guessed. The test set must also include hard cases, where staff themselves had to ask a manager. A test set made only of easy cases produces a flattering score that tells you nothing.

Step 2: One sentence for the role, one for the task

Anthropic says that setting a role in the system prompt helps the model focus on the right behaviour and tone for your use case, and that even a single sentence makes a difference. This step costs very little. Next comes the task part of Google’s framework: one sentence stating clearly what the output is.

The prompt below tells the model it is the marketplace’s refund handler, working strictly to internal procedure, and that its task is to read the request in the request tag and return a decision (duyet / tu_choi / chuyen_nguoi) with a short reason citing the relevant SOP step.

You are the marketplace's refund processing agent, and you follow the internal procedure exactly.
Task: read the request in the request tag and return one decision
(approve / reject / escalate) with a short reason, citing the relevant SOP step.

Check: run the step 1 test set straight away against this bare-bones version. The number you record now is your baseline, the reference point for every later edit.

Step 3: Keep the SOP’s order, add the word “because”

SOPs are usually already written as step 1, step 2, step 3, and that works in your favour. Anthropic recommends numbered lists when order or completing every step matters, along with stating the output format and constraints explicitly. So do not merge the steps into a paragraph.

Copying the rules across is not enough, though. According to Anthropic, explaining the reason behind an instruction helps the model understand your goal. Compare the two versions below. “Before” says: orders above the high-value threshold go to a human. “After” adds: because a wrong decision on this group of orders causes heavy losses and needs sign-off from someone with authority.

Before: Orders above the high-value threshold are escalated to a human.

After:  Orders above the high-value threshold are escalated to a human,
      because a wrong decision on this group of orders causes large losses
      and needs sign-off from someone with authority.

The second version tells the model what the rule is for. So when it meets a case the SOP did not foresee — say, a low-value order with signs of fraud — the model has grounds to escalate it rather than guess.

The reason behind each rule is usually not in the document but in the heads of the people who run the process, so you have to ask them.

Step 4: Use tags to separate rules, documents and input

A real SOP prompt usually holds several things at once: the main procedure, policy appendices, examples and the request to be handled. Anthropic recommends nesting XML tags when content has a natural hierarchy — for instance, documents inside a documents tag, each in its own document tag with an index attribute.

From here on, the sample blocks use square brackets instead of the angle brackets of XML tags for easier display; in a real prompt, use angle brackets. In the block below, document 1 is the refund SOP with numbered steps and reasons, document 2 is an appendix listing non-refundable items, the examples come in step 5, and the request tag holds the buyer’s request.

[documents]
  [document index="1"] Refund SOP, numbered steps with reasons [/document]
  [document index="2"] Appendix: list of non-refundable items [/document]
[/documents]
[examples]
  ... see step 5 ...
[/examples]
[request]
  ... the buyer's request ...
[/request]

Check: reread the prompt and ask whether any sentence sits outside a tag where the model might mistake it for input data. If so, move it to where it belongs.

Step 5: Turn the SOP’s exceptions into examples

Anthropic calls examples one of the most reliable ways to steer the format, tone and structure of output. Good examples must be close to the use case and diverse enough: they need to cover edge cases and differ enough from one another that the model does not pick up patterns you never meant to teach.

The “notes” and “special cases” sections of an SOP are a ready-made source of examples. Draw them from real historical cases, and wrap each one in an example tag so the model can tell examples from instructions. The example below describes a low-value order where the buyer sent three refund requests in one week; the decision is to escalate, citing step 5: an unusual request frequency needs human review.

[example]
  [request] Low-value order, buyer sent 3 refund requests in one week [/request]
  [decision] escalate [/decision]
  [reason] Step 5: an unusual request frequency needs a human to check. [/reason]
[/example]

Common mistake: all three examples end in “approve”. The model may conclude that approval is the safe answer. Spread the examples evenly across the decision branches. And do not reuse cases from the test set as examples, or the score will be artificially high.

Step 6: Give it to a colleague before you give it to the model

Anthropic has a “golden rule”: show the prompt to a colleague who knows almost nothing about the task and ask them to follow it. If they are confused, the model will be too. The test suits SOPs well, because SOPs often assume the reader has already absorbed the company’s internal culture.

Imagine the colleague asks: “What is the high-value threshold?” That is a number the client never wrote into the SOP. Ask the client, put the number in the prompt, then add a test case that sits right at the threshold.

Step 7: Measure, fix, and measure again when the model changes

Google’s course treats evaluating output and refining the prompt as the key to getting the results you want. In other words, an SOP prompt is not something you write once and finish. After every edit, rerun the whole test set and compare against the step 2 baseline.

Change one thing at a time so you know exactly which change moved the score up or down.

The work does not end at handover. Because model behaviour changes with each snapshot, every time the client upgrades the model your test set becomes a regression suite. Agree with the client from the outset that no one changes the production model until the test set passes.

Why the hardest part sits on the client’s side

Looking back over the seven steps, most of the raw material lives with the client: the reasons behind each rule, past cases with answers, thresholds nobody wrote down. So when planning a real project, budget more time sitting with the operations staff than writing the wording.

The prompt is only a record of what you have understood about the client’s process.

A tip for reading FDE job descriptions: any line that mentions evals, working with operations teams or turning business processes into working systems is where this skill gets used. On your CV, do not write “proficient in prompt engineering”.

Be specific instead: how many historical cases you built the eval set from, how far accuracy rose from the baseline, and how you caught a regression when the model changed.

Clients will not remember how well your prompt was written. They will remember the day the vendor shipped a new model and your test set caught the error before it reached a buyer.