# In an LLM grade-and-fix loop, the hard part is knowing when to stop

> Anthropic's reference pattern stops on PASS but sets no cap on rounds, so any version that answers real customers needs exits of its own.

Original: https://fdetimes.net/en/guides/llm-evaluator-optimizer-loop-stop-conditions/

In Anthropic's sample notebook for the evaluator–optimizer pattern, the loop is a `while True:` that returns only when the evaluator says exactly `PASS`. There is no cap on the number of rounds. If the evaluator is never satisfied, the code simply keeps running.

In a notebook that is harmless. Once you build a system that drafts replies to a company's customers, it becomes a real problem: every round costs another model call and more waiting time, while the draft may not improve at all.

Anthropic itself writes that agent loops usually come with stop conditions, such as a maximum number of iterations, to keep control.

This guide builds that loop in Python, from scratch to the point where it knows when to stop. The result is a function that takes a customer request, returns a draft along with the reason it stopped, and keeps a history of every round that you can show the client.

## What you need before writing the first line

You need Python 3, an API account with a model provider and a set of sample emails. The examples below call the model through an empty `call_model` function. That part is left out deliberately: wire it to whichever SDK you use, since each has its own syntax.

The running example: a customer asks why their refund is late. A reply passes if it states the correct order number, gives a timeline, promises nothing beyond policy and runs to no more than 120 words.

## Step 1: try a single call first

Any loop multiplies the number of model calls. So before writing one, run a single prompt over 10 sample emails to find out whether the simplest approach is already good enough.

```python
def call_model(system: str, prompt: str) -> str:
    """Stub: wire this to your SDK and return the text."""
    raise NotImplementedError

def generate(task: str, feedback: str = "") -> str:
    system = "You draft replies to customers: concise and within policy."
    prompt = task if not feedback else f"{task}\n\nRevise based on this feedback:\n{feedback}"
    return call_model(system, prompt)
```

**Check:** read the 10 drafts and mark which criteria each one misses. If all 10 pass, stop here. You do not need a loop.

## Step 2: write a rubric a human could also grade with

According to Anthropic, this workflow works well when there are clear evaluation criteria. The sample notebook adds another signal: the model itself must be able to give meaningful feedback.

Test that directly on the emails that failed in step 1. Write one line of specific feedback yourself, call `generate` again and see whether the email improves. If human feedback cannot rescue a draft, an evaluator will struggle to do better.

```python
RUBRIC = """Grade the email against 4 criteria:
1. States the correct order number.
2. Gives a specific timeline for the refund.
3. Promises nothing beyond the policy provided.
4. No more than 120 words.
Return PASS only if all 4 criteria are met and there is no remaining feedback."""
```

The last line of the rubric matters most: it forces the evaluator to return `PASS` only when it has nothing left to criticize.

## Step 3: make the evaluator return JSON, not prose

The original notebook compares the verdict string with `PASS`. The version below extends it to three values, `PASS`, `NEEDS_IMPROVEMENT` and `FAIL`, and wraps them in JSON so code can extract the verdict without guessing at the meaning of prose.

There are two ways to do this. The most reliable is to turn on Claude's structured outputs, which use constrained decoding to guarantee the result matches the schema, so malformed output no longer happens.

Note that it supports only part of JSON Schema, and numeric constraints such as `minimum`/`maximum` will make the API return a 400 error. A simple schema, with an enum for the verdict and an array of strings for the issues, is enough.

The code below takes the second route, trimmed down to run with any SDK: describe the JSON structure in the prompt and parse it yourself. The `except` branch only matters if you have not enabled structured outputs, or your provider has no equivalent feature; once they are on, you can drop it.

```python
import json

def evaluate(task: str, draft: str) -> dict:
    system = RUBRIC + '\nReturn JSON: {"verdict": "PASS|NEEDS_IMPROVEMENT|FAIL", "issues": ["..."]}'
    raw = call_model(system, f"Customer request:\n{task}\n\nDraft:\n{draft}")
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return {"verdict": "FAIL", "issues": ["evaluator returned unreadable output"]}
```

**Check:** print the `verdict` for all 10 emails. When you parse it yourself, broken JSON lands in the `except` branch and the email is handed to a human instead of crashing the loop.

## Step 4: add hard checks the model cannot argue with

Research on whether models can correct themselves remains unsettled. Self-Refine, by Madaan and colleagues, showed that an LLM giving feedback on its own output and then revising it can improve over several rounds.

But the study "Large Language Models Cannot Self-Correct Reasoning Yet" found that LLMs struggle to correct their answers without external feedback, and sometimes make them worse.

Judges have biases too. Another study recorded significant self-preference in GPT-4 when grading, and found that LLMs score text familiar to them, measured by low perplexity, markedly higher than humans do.

So if a criterion can be measured in code, let code measure it. Criteria 1 and 4 in the example fall into that category:

```python
def hard_checks(draft: str, order_id: str) -> list[str]:
    issues = []
    if order_id not in draft:
        issues.append(f"Missing order number {order_id}.")
    if len(draft.split()) > 120:
        issues.append("Email is longer than 120 words.")
    return issues
```

For the other two criteria, you can use a different model as the grader, or at least compare its scores with human scores on a few dozen emails.

## Step 5: four ways for the loop to stop

This is the part the original pattern leaves open. A 2026 preprint treats choosing when to stop as an optimal stopping problem: keep revising only while the expected improvement is worth the cost. The function below is a practical version of that idea, with four exits: pass, hand off to a human, stalled and out of rounds.

```python
MAX_ROUNDS = 3

def run(task: str, order_id: str):
    feedback, history = "", []
    for round_no in range(1, MAX_ROUNDS + 1):
        draft = generate(task, feedback)
        verdict = evaluate(task, draft)
        hard = hard_checks(draft, order_id)
        issues = hard + verdict.get("issues", [])
        history.append({"round": round_no, "draft": draft, "issues": issues})
        if verdict.get("verdict") == "PASS" and not hard:
            return draft, "pass", history
        if verdict.get("verdict") == "FAIL":
            return draft, "escalate", history
        if len(history) > 1 and issues == history[-2]["issues"]:
            return draft, "stalled", history
        feedback = "\n".join(issues)
    return draft, "max_rounds", history
```

The cost is easy to work out: with `MAX_ROUNDS = 3`, the worst case is 3 writes plus 3 grades, or 6 model calls per email. The `stalled` branch is the simplest way to predict that the next round will add nothing: if the issues are identical to the last round, further revision only costs money.

Treating `FAIL` as a handoff to a human is a design choice specific to this guide, since the original pattern exits only on `PASS`. It fits what Anthropic describes: agents can pause for human feedback at checkpoints or when they hit a blocker. An email judged broken at the root cannot be fixed by rewording.

**Key point:** A self-correction loop without a stopping point is just an invoice that has not come due.

**Check:** rerun the 10 sample emails and count how many end on each stop reason. If most finish at `max_rounds`, the problem usually lies in the rubric or the drafting prompt, not in a shortage of rounds.

## Common mistakes

The most common mistake is a vague rubric such as "the email must be polite and professional". The evaluator gives generic feedback, the draft keeps changing words without getting better, and the loop runs until it is out of rounds.

The second is letting the model's PASS override the hard checks: in the code above, an email missing the order number can never exit through the `pass` branch.

The third is throwing away `history`. That list tells you which criterion costs the loop the most rounds, and it is the best data you have for fixing the prompt.

## What this skill looks like at a client

On a real project, you do not invent the rubric yourself. You sit down with the head of customer support, collect the emails their team has had to correct, and turn each reason for a correction into a criterion that can be verified. That is genuine customer discovery; the difference is that what you bring back is a Python function.

When you present, bring a tally of stop reasons rather than a few polished drafts. The numbers below only illustrate how to read the table, following on from the guide's 10 sample emails; they are not real measurements:

| Stop reason | Emails | Model calls per email | Total calls |
|---|---|---|---|
| pass in round 1 | 6 | 2 | 12 |
| pass in round 2 | 2 | 4 | 8 |
| stalled in round 2 | 1 | 4 | 4 |
| escalate in round 1 | 1 | 2 | 2 |

From this table the client sees three things at once: 8 of 10 emails got through on their own, 2 reached a person along with their feedback history, and the whole batch took 26 calls against a ceiling of 60. The `stalled` email is where it pays to open `history`: if it got stuck on a missing timeline, the refund policy may never have made it into the prompt.

If you want to put this on your CV, do not write "built a self-correcting agent". Write that you designed a grade-and-fix loop with a cap on rounds and a human handoff, and include the share of emails that passed in the first round. In interviews, be ready to explain the system's limits: when it stops, and who takes over when it does.

The client will not ask how well your model revises emails. They will ask what happens when it cannot, and your `run` function needs to have the answer ready.

**Try this week:**

- Take 10 real customer replies, run the prompt once and record which criteria fail before you write the loop
- Write a 4-criterion rubric that can be verified, and a hard_checks function for at least two of those criteria
- Run the loop with MAX_ROUNDS = 3 and tabulate how many emails stop at pass, stalled, escalate and max_rounds, with the total number of model calls

## Sources

- [Building effective agents (Anthropic Engineering)](https://www.anthropic.com/engineering/building-effective-agents)

- [evaluator_optimizer.ipynb (anthropics/claude-cookbooks, patterns/agents)](https://raw.githubusercontent.com/anthropics/claude-cookbooks/refs/heads/main/patterns/agents/evaluator_optimizer.ipynb)

- [Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798)

- [Self-Preference Bias in LLM-as-a-Judge](https://arxiv.org/abs/2410.21819)

- [Structured outputs (Claude Docs)](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://arxiv.org/abs/2303.17651)

- [Optimal Stopping of Self-Refining Foundation Models](https://arxiv.org/abs/2608.10729)
