Stop asking the big model what to do after every step
Splitting a loan-case agent into planner, DAG check, executor, critic and replan makes it measurable, though a fixed process needs only a hand-written workflow.
In brief
- The planner produces the whole plan in one call and the executor runs each step with tools, so the large model is not consulted after every action.
- The plan must be validated as a DAG, with no unknown dependencies and no cycles, then run with retries and explicit trigger rules.
- The critic grades the draft against a checklist, but it needs a cap on rounds and a way to hand the case to a person.
In classic ReAct, the LLM plans only one sub-problem at a time. That works for a three-step task.
A loan application is different. It needs data extraction, an identity document check, income verification, a risk score and a drafted reply. An agent built this way has to go back to the large model after every action, and you cannot say in advance which path it will take.
The plan-and-execute architecture, described by LangChain in its post on planning agents, splits the job in two. A planner uses an LLM to generate a multi-step plan for the whole task. An executor receives the user’s question along with one step of the plan, then calls one or more tools to complete that step.
This guide builds such a case-processing agent in five steps: planner, DAG check, executor with retries, critic, and replanning.
What do you need?
You need Python 3, a function call_llm(prompt) -> str that wraps whichever model you use, and a TOOLS dict that maps tool names to Python functions. The code below is simplified for readability: there is no timeout handling, no logging and no full schema validation. The loan application is a hypothetical example; swap in your own domain. The prompts and error messages in the code are in Vietnamese, as in the original; translate them for your own use.
One piece of advice from Anthropic’s “Building effective agents” is worth keeping in mind before you start: find the simplest solution possible and only add complexity when it is genuinely needed. If the client’s process is already fixed, a hand-written workflow is enough. A planner only earns its place when the order of steps changes from case to case.
Step 1: The planner must return data, not prose
Make the planner return JSON with a depends_on field. That field turns the plan into something a machine can check.
import json
PLAN_PROMPT = """You are the planner. For the case file below, return JSON:
{"steps": [{"id": "...", "task": "...", "tool": "...", "depends_on": ["..."]}]}
Only use these tools: extract_fields, check_id, verify_income, score_risk, draft_reply.
Case file:
"""
def plan(case_text):
raw = call_llm(PLAN_PROMPT + case_text)
steps = json.loads(raw)["steps"]
for s in steps:
if s["tool"] not in TOOLS:
raise ValueError(f"Invalid tool: {s['tool']}")
return steps
Check: print the plans for three different cases. check_id and verify_income should both depend on extract_fields but not on each other. If the planner arranges them in a single chain, your prompt has not made clear that the two tasks are independent.
Step 2: Do not trust a plan until you have checked it is a DAG
The Airflow documentation defines a DAG as the model that holds everything needed to run a workflow. Its structure comes from the dependencies declared between tasks. An LLM-generated plan is the same kind of dependency declaration, except that it can be wrong.
def check_refs(steps):
ids = {s["id"] for s in steps}
unknown = {d for s in steps for d in s["depends_on"]} - ids
if unknown:
raise ValueError(f"Depends on steps that do not exist: {unknown}")
def topo_order(steps):
check_refs(steps)
deps = {s["id"]: set(s["depends_on"]) for s in steps}
order = []
while deps:
ready = [k for k, ds in deps.items() if not ds]
if not ready:
raise ValueError(f"Plan contains a cycle: {list(deps)}")
for k in ready:
order.append(k)
del deps[k]
for ds in deps.values():
ds.difference_update(ready)
return order
To see what the code actually produces, imagine the planner returns the following plan for a sample loan application (an illustration, not real output from any model). The steps are: extract key fields, check the national ID card, check the bank statement, score the risk, and draft the reply letter.
{"steps": [
{"id": "s1", "task": "Extract key fields", "tool": "extract_fields", "depends_on": []},
{"id": "s2", "task": "Verify national ID card", "tool": "check_id", "depends_on": ["s1"]},
{"id": "s3", "task": "Check bank statements", "tool": "verify_income", "depends_on": ["s1"]},
{"id": "s4", "task": "Score risk", "tool": "score_risk", "depends_on": ["s2", "s3"]},
{"id": "s5", "task": "Draft reply letter", "tool": "draft_reply", "depends_on": ["s4"]}
]}
Call the function on this steps list:
>>> topo_order(steps)
['s1', 's2', 's3', 's4', 's5']
The while loop runs four times. On the first pass only s1 is ready. On the second, s2 and s3 are ready together, so those two steps can run in parallel. The third pass reaches s4, and the last one s5.
Check: edit the plan above by hand so that s4 depends on s5. The function must raise Plan contains a cycle: ['s4', 's5'] (“the plan contains a cycle”) rather than hang. An error at this stage should go back to the planner with the error message, not carry on into execution.
Step 3: The executor needs retries and explicit run rules
Two Airflow ideas apply directly here. The first is setting default parameters such as retries once for every step. The second is the trigger rule: the default, all_success, means a step runs only when every upstream step has succeeded, while all_done lets a step run once upstream steps have finished, even if they failed.
DEFAULTS = {"retries": 2}
def run_step(step, case_text, results):
inputs = {d: results.get(d) for d in step["depends_on"]}
last_err = None
for _ in range(DEFAULTS["retries"] + 1):
try:
return TOOLS[step["tool"]](case_text, step["task"], inputs)
except Exception as e:
last_err = e
raise last_err
def execute(steps, case_text):
by_id = {s["id"]: s for s in steps}
results, status = {}, {}
for sid in topo_order(steps):
step = by_id[sid]
rule = step.get("trigger_rule", "all_success")
upstream_ok = all(status[d] == "success" for d in step["depends_on"])
if rule == "all_success" and not upstream_ok:
status[sid] = "skipped"
continue
try:
results[sid] = run_step(step, case_text, results)
status[sid] = "success"
except Exception:
status[sid] = "failed"
return results, status
This executor does not call the large model between steps. LangChain treats that as the architecture’s main benefit: multi-step workflows run faster because the large agent does not have to be consulted after every action.
Check: make verify_income always raise an error. With the sample plan from Step 2, the correct result is s3 failed, with s4 and s5 skipped. A step such as “write the review log” should be set to all_done, so that a record survives wherever the case breaks.
Step 4: The critic grades the work; it does not rewrite it
According to LangChain, reflection is a prompting strategy that improves the quality and success rate of agents. In a generator/critic loop, the critic is prompted to act as a teacher and give constructive feedback on the first answer.
Anthropic calls this pattern evaluator-optimizer: one LLM call produces a result, another evaluates it and gives feedback, and the two repeat in a loop.
CRITIC_PROMPT = """You are a strict grader. Check the draft against the evidence.
Checklist: amounts match the documents; no promises the policy does not allow;
every reason for rejection is backed by evidence. Return JSON {"pass": true/false, "issues": [...]}.
"""
def reflect(draft, evidence, max_rounds=2):
verdict = None
for _ in range(max_rounds):
payload = json.dumps({"draft": draft, "evidence": evidence}, ensure_ascii=False)
verdict = json.loads(call_llm(CRITIC_PROMPT + payload))
if verdict["pass"]:
return draft, verdict
draft = call_llm(f"Revise the draft based on this feedback {verdict['issues']}:\n{draft}")
return draft, verdict
The prompt asks the critic to act as a strict teacher and check three things: amounts match the documents, the draft promises nothing the policy does not allow, and every reason for rejection is backed by evidence.
The checklist is where the value lies. A critic asked only a vague “is this good?” has no criteria to work from. Three specific criteria, taken from the client’s own rules, give it something to check against and give you a clear reason every time a draft is sent back.
Step 5: Replan, and know when to stop
LangChain describes how, after execution, the agent is called again with a replan prompt to choose between answering now and generating a further plan. That loop needs a ceiling.
REPLAN_PROMPT = """Based on the status of the steps, return JSON:
{"action": "finish", "response": "..."} or {"action": "replan", "steps": [...]}
"""
def handle_case(case_text, max_replans=2):
steps = plan(case_text)
for _ in range(max_replans + 1):
results, status = execute(steps, case_text)
payload = json.dumps({"status": status, "results": results}, ensure_ascii=False, default=str)
decision = json.loads(call_llm(REPLAN_PROMPT + payload))
if decision["action"] == "finish":
return reflect(decision["response"], results)
steps = decision["steps"]
return {"action": "escalate_to_human", "status": status}
The escalate_to_human branch is a design choice, not a minor detail. In client work, a case handed to a reviewer along with the status of every step is far easier to accept than a confident answer that is wrong.
Common mistakes
The first mistake is letting the critic read only the draft without seeing the original evidence. If extract_fields misreads an amount, the draft can still look consistent, and the critic has nothing to check it against.
LangChain notes a related limitation of Reflexion: it follows a single fixed trajectory, so one wrong step can affect the decisions that follow. That is why the original evidence goes into the critic, as in the code above, so that an early error still has a chance of being caught.
The second mistake is dropping ReAct entirely. According to the ReAct authors, interleaving reasoning with action helps the model track and update its plan and handle exceptions. For a hard step, such as reading a bank statement in an unusual format, the executor can perfectly well be a small ReAct agent running inside one node of the DAG.
The third mistake is using the same retries setting for every step. Sending an email to the customer should not be retried the way reading a file is, because a retry there could mean the customer gets three emails.
Where does this skill show up on a client site?
On your first day with a client, the first job is not writing prompts. Sit down with the reviewers and draw their real process as a DAG: which step waits for which, which failure stops the whole case, which failure still has to be recorded. That drawing becomes depends_on and trigger_rule, and their checklist becomes the critic’s prompt.
When reading job descriptions for FDE or AI engineer roles, look for phrases such as “multi-step workflows”, “orchestration” and “evaluation”. On your CV, do not just write “built an agent with an LLM”. Say that you separated the planner from the executor, validated plans as a DAG, and state what percentage of cases the system handed to a person.
A good agent for multi-step cases is not one that can do everything by itself. It is one where, when it breaks, you can point to the node that failed and explain why.
Was this article useful?
Thanks for the feedback!