# Stop asking the big model what to do after every step

> Splitting a loan-case agent into planner, DAG check, executor, critic and replan makes it measurable, though a fixed process needs only a hand-written workflow.

Original: https://fdetimes.net/en/guides/planner-executor-agent-dag-critic-loop/

In classic ReAct, the LLM plans only one sub-problem at a time. That works for a three-step task.

A loan application is different. It needs data extraction, an identity document check, income verification, a risk score and a drafted reply. An agent built this way has to go back to the large model after every action, and you cannot say in advance which path it will take.

The plan-and-execute architecture, described by LangChain in its post on planning agents, splits the job in two. A planner uses an LLM to generate a multi-step plan for the whole task. An executor receives the user's question along with one step of the plan, then calls one or more tools to complete that step.

This guide builds such a case-processing agent in five steps: planner, DAG check, executor with retries, critic, and replanning.

## What do you need?

You need Python 3, a function `call_llm(prompt) -> str` that wraps whichever model you use, and a `TOOLS` dict that maps tool names to Python functions. The code below is **simplified** for readability: there is no timeout handling, no logging and no full schema validation. The loan application is a hypothetical example; swap in your own domain. The prompts and error messages in the code are in Vietnamese, as in the original; translate them for your own use.

One piece of advice from Anthropic's "Building effective agents" is worth keeping in mind before you start: find the simplest solution possible and only add complexity when it is genuinely needed. If the client's process is already fixed, a hand-written workflow is enough. A planner only earns its place when the order of steps changes from case to case.

## Step 1: The planner must return data, not prose

Make the planner return JSON with a `depends_on` field. That field turns the plan into something a machine can check.

```python
import json

PLAN_PROMPT = """You are the planner. For the case file below, return JSON:
{"steps": [{"id": "...", "task": "...", "tool": "...", "depends_on": ["..."]}]}
Only use these tools: extract_fields, check_id, verify_income, score_risk, draft_reply.
Case file:
"""

def plan(case_text):
    raw = call_llm(PLAN_PROMPT + case_text)
    steps = json.loads(raw)["steps"]
    for s in steps:
        if s["tool"] not in TOOLS:
            raise ValueError(f"Invalid tool: {s['tool']}")
    return steps
```

**Check:** print the plans for three different cases. `check_id` and `verify_income` should both depend on `extract_fields` but not on each other. If the planner arranges them in a single chain, your prompt has not made clear that the two tasks are independent.

## Step 2: Do not trust a plan until you have checked it is a DAG

The Airflow documentation defines a DAG as the model that holds everything needed to run a workflow. Its structure comes from the dependencies declared between tasks. An LLM-generated plan is the same kind of dependency declaration, except that it can be wrong.

```python
def check_refs(steps):
    ids = {s["id"] for s in steps}
    unknown = {d for s in steps for d in s["depends_on"]} - ids
    if unknown:
        raise ValueError(f"Depends on steps that do not exist: {unknown}")

def topo_order(steps):
    check_refs(steps)
    deps = {s["id"]: set(s["depends_on"]) for s in steps}
    order = []
    while deps:
        ready = [k for k, ds in deps.items() if not ds]
        if not ready:
            raise ValueError(f"Plan contains a cycle: {list(deps)}")
        for k in ready:
            order.append(k)
            del deps[k]
        for ds in deps.values():
            ds.difference_update(ready)
    return order
```

To see what the code actually produces, imagine the planner returns the following plan for a sample loan application (an illustration, not real output from any model). The steps are: extract key fields, check the national ID card, check the bank statement, score the risk, and draft the reply letter.

```json
{"steps": [
 {"id": "s1", "task": "Extract key fields", "tool": "extract_fields", "depends_on": []},
 {"id": "s2", "task": "Verify national ID card", "tool": "check_id", "depends_on": ["s1"]},
 {"id": "s3", "task": "Check bank statements", "tool": "verify_income", "depends_on": ["s1"]},
 {"id": "s4", "task": "Score risk", "tool": "score_risk", "depends_on": ["s2", "s3"]},
 {"id": "s5", "task": "Draft reply letter", "tool": "draft_reply", "depends_on": ["s4"]}
]}
```

Call the function on this `steps` list:

```
>>> topo_order(steps)
['s1', 's2', 's3', 's4', 's5']
```

The `while` loop runs four times. On the first pass only `s1` is ready. On the second, `s2` and `s3` are ready together, so those two steps can run in parallel. The third pass reaches `s4`, and the last one `s5`.

**Check:** edit the plan above by hand so that `s4` depends on `s5`. The function must raise `Plan contains a cycle: ['s4', 's5']` ("the plan contains a cycle") rather than hang. An error at this stage should go back to the planner with the error message, not carry on into execution.

## Step 3: The executor needs retries and explicit run rules

Two Airflow ideas apply directly here. The first is setting default parameters such as `retries` once for every step. The second is the trigger rule: the default, `all_success`, means a step runs only when every upstream step has succeeded, while `all_done` lets a step run once upstream steps have finished, even if they failed.

```python
DEFAULTS = {"retries": 2}

def run_step(step, case_text, results):
    inputs = {d: results.get(d) for d in step["depends_on"]}
    last_err = None
    for _ in range(DEFAULTS["retries"] + 1):
        try:
            return TOOLS[step["tool"]](case_text, step["task"], inputs)
        except Exception as e:
            last_err = e
    raise last_err

def execute(steps, case_text):
    by_id = {s["id"]: s for s in steps}
    results, status = {}, {}
    for sid in topo_order(steps):
        step = by_id[sid]
        rule = step.get("trigger_rule", "all_success")
        upstream_ok = all(status[d] == "success" for d in step["depends_on"])
        if rule == "all_success" and not upstream_ok:
            status[sid] = "skipped"
            continue
        try:
            results[sid] = run_step(step, case_text, results)
            status[sid] = "success"
        except Exception:
            status[sid] = "failed"
    return results, status
```

This executor does not call the large model between steps. LangChain treats that as the architecture's main benefit: multi-step workflows run faster because the large agent does not have to be consulted after every action.

**Check:** make `verify_income` always raise an error. With the sample plan from Step 2, the correct result is `s3` `failed`, with `s4` and `s5` `skipped`. A step such as "write the review log" should be set to `all_done`, so that a record survives wherever the case breaks.

## Step 4: The critic grades the work; it does not rewrite it

According to LangChain, reflection is a prompting strategy that improves the quality and success rate of agents. In a generator/critic loop, the critic is prompted to act as a teacher and give constructive feedback on the first answer.

Anthropic calls this pattern evaluator-optimizer: one LLM call produces a result, another evaluates it and gives feedback, and the two repeat in a loop.

```python
CRITIC_PROMPT = """You are a strict grader. Check the draft against the evidence.
Checklist: amounts match the documents; no promises the policy does not allow;
every reason for rejection is backed by evidence. Return JSON {"pass": true/false, "issues": [...]}.
"""

def reflect(draft, evidence, max_rounds=2):
    verdict = None
    for _ in range(max_rounds):
        payload = json.dumps({"draft": draft, "evidence": evidence}, ensure_ascii=False)
        verdict = json.loads(call_llm(CRITIC_PROMPT + payload))
        if verdict["pass"]:
            return draft, verdict
        draft = call_llm(f"Revise the draft based on this feedback {verdict['issues']}:\n{draft}")
    return draft, verdict
```

The prompt asks the critic to act as a strict teacher and check three things: amounts match the documents, the draft promises nothing the policy does not allow, and every reason for rejection is backed by evidence.

The checklist is where the value lies. A critic asked only a vague "is this good?" has no criteria to work from. Three specific criteria, taken from the client's own rules, give it something to check against and give you a clear reason every time a draft is sent back.

## Step 5: Replan, and know when to stop

LangChain describes how, after execution, the agent is called again with a replan prompt to choose between answering now and generating a further plan. That loop needs a ceiling.

```python
REPLAN_PROMPT = """Based on the status of the steps, return JSON:
{"action": "finish", "response": "..."} or {"action": "replan", "steps": [...]}
"""

def handle_case(case_text, max_replans=2):
    steps = plan(case_text)
    for _ in range(max_replans + 1):
        results, status = execute(steps, case_text)
        payload = json.dumps({"status": status, "results": results}, ensure_ascii=False, default=str)
        decision = json.loads(call_llm(REPLAN_PROMPT + payload))
        if decision["action"] == "finish":
            return reflect(decision["response"], results)
        steps = decision["steps"]
    return {"action": "escalate_to_human", "status": status}
```

The `escalate_to_human` branch is a design choice, not a minor detail. In client work, a case handed to a reviewer along with the status of every step is far easier to accept than a confident answer that is wrong.

**Key point:** An LLM-generated plan is only a proposal. It becomes a program once it has passed a DAG check and has retries, trigger rules and a cap on rounds.

## Common mistakes

The first mistake is letting the critic read only the draft without seeing the original evidence. If `extract_fields` misreads an amount, the draft can still look consistent, and the critic has nothing to check it against.

LangChain notes a related limitation of Reflexion: it follows a single fixed trajectory, so one wrong step can affect the decisions that follow. That is why the original evidence goes into the critic, as in the code above, so that an early error still has a chance of being caught.

The second mistake is dropping ReAct entirely. According to the ReAct authors, interleaving reasoning with action helps the model track and update its plan and handle exceptions. For a hard step, such as reading a bank statement in an unusual format, the executor can perfectly well be a small ReAct agent running inside one node of the DAG.

The third mistake is using the same `retries` setting for every step. Sending an email to the customer should not be retried the way reading a file is, because a retry there could mean the customer gets three emails.

## Where does this skill show up on a client site?

On your first day with a client, the first job is not writing prompts. Sit down with the reviewers and draw their real process as a DAG: which step waits for which, which failure stops the whole case, which failure still has to be recorded. That drawing becomes `depends_on` and `trigger_rule`, and their checklist becomes the critic's prompt.

When reading job descriptions for FDE or AI engineer roles, look for phrases such as "multi-step workflows", "orchestration" and "evaluation". On your CV, do not just write "built an agent with an LLM". Say that you separated the planner from the executor, validated plans as a DAG, and state what percentage of cases the system handed to a person.

A good agent for multi-step cases is not one that can do everything by itself. It is one where, when it breaks, you can point to the node that failed and explain why.

**Try this week:**

- Take a real multi-step process at your company, draw it as a DAG on paper, then get the planner to reproduce that exact DAG from a written description.
- Write the topo_order function from this guide and test it on three broken plans: a dependency on a step that does not exist, two steps that depend on each other in a cycle, and a tool that is not on the list.
- Run 10 sample cases, record the number of critic rounds and replans for each, and set your stopping thresholds based on those numbers.

## Sources

- [Plan-and-Execute Agents (LangChain)](https://www.langchain.com/blog/planning-agents)

- [Dags (Airflow Documentation)](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html)

- [Reflection Agents (LangChain)](https://www.langchain.com/blog/reflection-agents)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://react-lm.github.io/)

- [Building effective agents (Anthropic)](https://www.anthropic.com/engineering/building-effective-agents)
