FDE Times dataFDE jobs open 873▼9 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 55Companies hiring 181Remote-friendly 25%Median US pay $200kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Guides

In an LLM grade-and-fix loop, the hard part is knowing when to stop

Anthropic's reference pattern stops on PASS but sets no cap on rounds, so any version that answers real customers needs exits of its own.

ShareLinkedInFacebookX

In brief

  • Anthropic's cookbook pattern returns on PASS but has no cap on rounds, so a production version has to add one.
  • An evaluator only helps when the criteria are clear; evidence on model self-correction is mixed, so let code measure whatever can be measured.
  • Try a single prompt first, and build the loop only when the simple approach falls short.
Loop diagram with four steps: Draft, Score, Hard checks, Notes, then back to Draft. At the centre of the loop is a large orange 3 labelled "rounds max". From the Score step, a branch splits into four exits: Pass (all 4 criteria met), Escalate (verdict FAIL), Stalled (repeat notes) and Cap hit. Cap hit is highlighted in orange for emphasis.
The run function loops up to three times, drafting a letter, scoring it, applying hard checks and giving notes, then leaves by one of four exits. The "cap hit" exit is what Anthropic's evaluator–optimizer pattern lacks. Source: Anthropic's evaluator–optimizer notebook and the example code in the article.

In Anthropic’s sample notebook for the evaluator–optimizer pattern, the loop is a while True: that returns only when the evaluator says exactly PASS. There is no cap on the number of rounds. If the evaluator is never satisfied, the code simply keeps running.

In a notebook that is harmless. Once you build a system that drafts replies to a company’s customers, it becomes a real problem: every round costs another model call and more waiting time, while the draft may not improve at all.

Anthropic itself writes that agent loops usually come with stop conditions, such as a maximum number of iterations, to keep control.

This guide builds that loop in Python, from scratch to the point where it knows when to stop. The result is a function that takes a customer request, returns a draft along with the reason it stopped, and keeps a history of every round that you can show the client.

What you need before writing the first line

You need Python 3, an API account with a model provider and a set of sample emails. The examples below call the model through an empty call_model function. That part is left out deliberately: wire it to whichever SDK you use, since each has its own syntax.

The running example: a customer asks why their refund is late. A reply passes if it states the correct order number, gives a timeline, promises nothing beyond policy and runs to no more than 120 words.

Step 1: try a single call first

Any loop multiplies the number of model calls. So before writing one, run a single prompt over 10 sample emails to find out whether the simplest approach is already good enough.

def call_model(system: str, prompt: str) -> str:
    """Stub: wire this to your SDK and return the text."""
    raise NotImplementedError

def generate(task: str, feedback: str = "") -> str:
    system = "You draft replies to customers: concise and within policy."
    prompt = task if not feedback else f"{task}\n\nRevise based on this feedback:\n{feedback}"
    return call_model(system, prompt)

Check: read the 10 drafts and mark which criteria each one misses. If all 10 pass, stop here. You do not need a loop.

Step 2: write a rubric a human could also grade with

According to Anthropic, this workflow works well when there are clear evaluation criteria. The sample notebook adds another signal: the model itself must be able to give meaningful feedback.

Test that directly on the emails that failed in step 1. Write one line of specific feedback yourself, call generate again and see whether the email improves. If human feedback cannot rescue a draft, an evaluator will struggle to do better.

RUBRIC = """Grade the email against 4 criteria:
1. States the correct order number.
2. Gives a specific timeline for the refund.
3. Promises nothing beyond the policy provided.
4. No more than 120 words.
Return PASS only if all 4 criteria are met and there is no remaining feedback."""

The last line of the rubric matters most: it forces the evaluator to return PASS only when it has nothing left to criticize.

Step 3: make the evaluator return JSON, not prose

The original notebook compares the verdict string with PASS. The version below extends it to three values, PASS, NEEDS_IMPROVEMENT and FAIL, and wraps them in JSON so code can extract the verdict without guessing at the meaning of prose.

There are two ways to do this. The most reliable is to turn on Claude’s structured outputs, which use constrained decoding to guarantee the result matches the schema, so malformed output no longer happens.

Note that it supports only part of JSON Schema, and numeric constraints such as minimum/maximum will make the API return a 400 error. A simple schema, with an enum for the verdict and an array of strings for the issues, is enough.

The code below takes the second route, trimmed down to run with any SDK: describe the JSON structure in the prompt and parse it yourself. The except branch only matters if you have not enabled structured outputs, or your provider has no equivalent feature; once they are on, you can drop it.

import json

def evaluate(task: str, draft: str) -> dict:
    system = RUBRIC + '\nReturn JSON: {"verdict": "PASS|NEEDS_IMPROVEMENT|FAIL", "issues": ["..."]}'
    raw = call_model(system, f"Customer request:\n{task}\n\nDraft:\n{draft}")
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return {"verdict": "FAIL", "issues": ["evaluator returned unreadable output"]}

Check: print the verdict for all 10 emails. When you parse it yourself, broken JSON lands in the except branch and the email is handed to a human instead of crashing the loop.

Step 4: add hard checks the model cannot argue with

Research on whether models can correct themselves remains unsettled. Self-Refine, by Madaan and colleagues, showed that an LLM giving feedback on its own output and then revising it can improve over several rounds.

But the study “Large Language Models Cannot Self-Correct Reasoning Yet” found that LLMs struggle to correct their answers without external feedback, and sometimes make them worse.

Judges have biases too. Another study recorded significant self-preference in GPT-4 when grading, and found that LLMs score text familiar to them, measured by low perplexity, markedly higher than humans do.

So if a criterion can be measured in code, let code measure it. Criteria 1 and 4 in the example fall into that category:

def hard_checks(draft: str, order_id: str) -> list[str]:
    issues = []
    if order_id not in draft:
        issues.append(f"Missing order number {order_id}.")
    if len(draft.split()) > 120:
        issues.append("Email is longer than 120 words.")
    return issues

For the other two criteria, you can use a different model as the grader, or at least compare its scores with human scores on a few dozen emails.

Step 5: four ways for the loop to stop

This is the part the original pattern leaves open. A 2026 preprint treats choosing when to stop as an optimal stopping problem: keep revising only while the expected improvement is worth the cost. The function below is a practical version of that idea, with four exits: pass, hand off to a human, stalled and out of rounds.

MAX_ROUNDS = 3

def run(task: str, order_id: str):
    feedback, history = "", []
    for round_no in range(1, MAX_ROUNDS + 1):
        draft = generate(task, feedback)
        verdict = evaluate(task, draft)
        hard = hard_checks(draft, order_id)
        issues = hard + verdict.get("issues", [])
        history.append({"round": round_no, "draft": draft, "issues": issues})
        if verdict.get("verdict") == "PASS" and not hard:
            return draft, "pass", history
        if verdict.get("verdict") == "FAIL":
            return draft, "escalate", history
        if len(history) > 1 and issues == history[-2]["issues"]:
            return draft, "stalled", history
        feedback = "\n".join(issues)
    return draft, "max_rounds", history

The cost is easy to work out: with MAX_ROUNDS = 3, the worst case is 3 writes plus 3 grades, or 6 model calls per email. The stalled branch is the simplest way to predict that the next round will add nothing: if the issues are identical to the last round, further revision only costs money.

Treating FAIL as a handoff to a human is a design choice specific to this guide, since the original pattern exits only on PASS. It fits what Anthropic describes: agents can pause for human feedback at checkpoints or when they hit a blocker. An email judged broken at the root cannot be fixed by rewording.

Check: rerun the 10 sample emails and count how many end on each stop reason. If most finish at max_rounds, the problem usually lies in the rubric or the drafting prompt, not in a shortage of rounds.

Common mistakes

The most common mistake is a vague rubric such as “the email must be polite and professional”. The evaluator gives generic feedback, the draft keeps changing words without getting better, and the loop runs until it is out of rounds.

The second is letting the model’s PASS override the hard checks: in the code above, an email missing the order number can never exit through the pass branch.

The third is throwing away history. That list tells you which criterion costs the loop the most rounds, and it is the best data you have for fixing the prompt.

What this skill looks like at a client

On a real project, you do not invent the rubric yourself. You sit down with the head of customer support, collect the emails their team has had to correct, and turn each reason for a correction into a criterion that can be verified. That is genuine customer discovery; the difference is that what you bring back is a Python function.

When you present, bring a tally of stop reasons rather than a few polished drafts. The numbers below only illustrate how to read the table, following on from the guide’s 10 sample emails; they are not real measurements:

Stop reason Emails Model calls per email Total calls
pass in round 1 6 2 12
pass in round 2 2 4 8
stalled in round 2 1 4 4
escalate in round 1 1 2 2

From this table the client sees three things at once: 8 of 10 emails got through on their own, 2 reached a person along with their feedback history, and the whole batch took 26 calls against a ceiling of 60. The stalled email is where it pays to open history: if it got stuck on a missing timeline, the refund policy may never have made it into the prompt.

If you want to put this on your CV, do not write “built a self-correcting agent”. Write that you designed a grade-and-fix loop with a cap on rounds and a human handoff, and include the share of emails that passed in the first round. In interviews, be ready to explain the system’s limits: when it stops, and who takes over when it does.

The client will not ask how well your model revises emails. They will ask what happens when it cannot, and your run function needs to have the answer ready.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
7 sources
Read next on the roadmap · Stage 3: Applied AIGive an agent 80 tools and it picks the wrong one more oftenKeep a few core tools loaded, group the rest under consistent prefixes and let the agent search for them when a task calls for it.