FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

Why agents forget at step 60, and how compaction fixes it

A six-step loop, running from the lightest technique to the heaviest, decides which parts of an agent's history to keep before every model call.

ShareLinkedInFacebookX

In brief

  • Long context costs compute (which grows with the square of sequence length) and makes models worse at using information in the middle, so deliberate compression beats stuffing in the whole history.
  • Go from light to heavy: clear old tool results, cut while keeping tool call/result pairs intact, summarize incrementally, then write notes outside the context.
  • Any summary can lose details or invent new ones, so it needs to be checked with several metrics, not just one.
Decision tree: if the token count exceeds the TRIGGER threshold, first clear old tool results, keeping the 3 most recent; if still over the threshold, run compact, summarising older history and cutting at a safe point; every other branch goes straight to the model call.
The compaction loop runs from the lightest fix to the heaviest: clearing spent tool output is cheap and safe, while summarisation runs only when that is not enough to bring the token count down.

At step 60, an agent handling tickets for a logistics client calls an API it already called at step 5, because the earlier result has been cut off or buried under dozens of blocks of JSON. Take that as a hypothetical, but a familiar one. The model has not got worse. Its context has been badly managed.

Those dozens of JSON blocks do damage in two ways. Every model call has to carry them, and IBM explains that compute requirements grow with the square of sequence length.

Meanwhile the step 5 result sits exactly where the model reads worst. IBM calls this the “lost in the middle” phenomenon: models perform worse when they have to work carefully with information in the middle of a long context.

JSON that has already served its purpose is not harmless either. Redis warns that a context window can hold only a limited amount, and that irrelevant information pollutes the context and drags down the quality of an agent’s reasoning. So the way to keep an agent running for a long time is to choose what to keep, rather than cram in more history.

FDEs run into this all the time: an agent runs ten turns flawlessly in a demo environment, then starts forgetting three hours into a session in the client’s environment. The guide below builds a six-step compaction loop, from the lightest technique to the heaviest.

What you will build, and what you need

The end product is a function, manage_context(messages), that runs before every model call. You need Python, an agent that calls tools and stores its history as a list of {"role", "content"} messages, and a function that calls an LLM. The code below is a simplified version for learning: count_tokens and call_llm are placeholders, to be replaced with the tokenizer and SDK of the model you use.

Step 1: measure first, then compress

LangChain’s documentation describes the basic strategy as counting the tokens in the message history and trimming when it nears the limit. Set the trigger threshold below the real limit, so there is room left for the summarization call itself.

LIMIT = 100_000          # a threshold you choose, smaller than the real context window
TRIGGER = int(0.8 * LIMIT)

def need_compaction(messages):
    return count_tokens(messages) > TRIGGER

Check: print the token count after every turn. It usually jumps right after tool calls that return a lot of data, and those are the places to deal with first.

Step 2: clear old tool results, the cheapest and safest move

Anthropic places tool result clearing among the safest, lightest-touch forms of compaction: raw tool output that has already been used gets dropped. Once an agent has read 5,000 lines of JSON and drawn its conclusion, it almost never needs to read the original again.

def clear_old_tool_results(messages, keep_last=3):
    tool_idx = [i for i, m in enumerate(messages) if m["role"] == "tool"]
    for i in tool_idx[:-keep_last]:
        messages[i]["content"] = "[old tool result cleared; see notes if needed]"
    return messages

The detail to notice is that the function replaces the content instead of deleting the message. That keeps the structure of the history intact, in line with the warning in the next step. Check: rerun the same scenario and compare the token count before and after.

Step 3: cut without breaking the history

LangChain’s documentation points out that when you remove messages from state, the remaining history must still be valid; for example, a tool call and its tool result must stay together. If the cut falls in the middle of a pair, the history opens with an orphaned tool result and the next model call may fail.

def safe_cut_index(messages, cut):
    # push the cut point later until the history no longer opens with an orphaned tool result
    while len(messages) > cut and messages[cut]["role"] == "tool":
        cut += 1
    return cut

This is a simplified version. If one of your tool calls returns several results, pair them by the call’s ID instead of relying on the role alone.

Step 4: summarize, because plain trimming loses information

LangChain also acknowledges that dropping messages from the queue can lose information. That is why the library ships SummarizationMiddleware, which uses a chat model to summarize the history. In essence this is compaction as Anthropic defines it: summarize a conversation that is filling up, then continue from the summary.

COMPACT_PROMPT = """Summarize the earlier conversation so the agent can carry on.
KEEP VERBATIM: the user's goal, decisions already made, every ID/file name/figure,
unresolved errors, work in progress and the next step.
DROP: greetings, tool output already used, attempts that were abandoned."""

def compact(messages, old_summary, keep_recent=10):
    cut = safe_cut_index(messages, len(messages) - keep_recent)
    old, recent = messages[:cut], messages[cut:]
    summary = call_llm(COMPACT_PROMPT, previous=old_summary, new=old)
    head = {"role": "user", "content": "Summary of the previous session:\n" + summary}
    return [head] + recent, summary

The code reflects two design choices. First, the function takes old_summary and feeds in only the new material. Redis describes this approach as updating the summary incrementally whenever new data arrives, instead of rewriting it from scratch.

The second choice lies in the prompt, written along the lines Anthropic recommends: first maximize recall so the prompt misses nothing relevant, then improve precision by trimming the excess.

That order has a reason. Redis points out that the cost of summarizing is the possible loss of details that look meaningless now but matter later. A missing customer ID does far more harm than a few extra sentences.

What does the result look like? Go back to the logistics agent, with made-up figures for illustration. The history stands at 85,000 tokens, over the 80,000 TRIGGER threshold from step 1. Step 2 is not enough to bring it down, so compact runs and the start of the context is replaced with a block like this:

Summary of the previous session:
- Goal: resolve the customer's late-delivery ticket, reply before end of day
- Decided: no refund, offer free redelivery
- IDs: order ORD-1042, waybill VD-88317
- Unresolved error: the waybill lookup API has timed out twice
- Next step: retry the lookup API, then draft the email to the customer

These five lines stand in for dozens of turns of conversation and tool output. What to check is that the IDs survive intact: if the summary says “a delayed order” instead of ORD-1042, the agent will call the API again, just as it did at step 60 in the opening example. Check: print the token count right after compact and read the summary block yourself, at least the first few times.

Step 5: move important state outside the context

Some things should not depend on the quality of a summary. Anthropic calls this technique structured note-taking, or agentic memory: the agent periodically writes notes to a memory store outside the context window.

def write_note(key, value, path="agent_notes.md"):
    with open(path, "a", encoding="utf-8") as f:
        f.write(f"- {key}: {value}\n")

Register this function as a tool, then instruct the agent in the system prompt to record every decision it settles. The placeholder from step 2 points to this file, so the agent knows where to look.

Heavy branches of work, such as reading an entire log archive, can be handed to a sub-agent. As Anthropic describes it, a sub-agent works in a clean context and sends only a condensed summary back to the main agent.

Step 6: the summary needs testing too

Stopping at step 5 and assuming the summary is fine is a risk, because the summarization step itself can drop details or invent them. The SEI team at Carnegie Mellon argues that evaluation with a single metric has not been shown to be effective, so current practice is to use a whole set of metrics.

SEI also notes that many accuracy metrics require a reference summary, which is often unavailable, while faithfulness metrics catch some of the fabrication errors.

Below is a minimal test suite (a suggestion, not a standard):

def eval_summary(summary, must_keep, source_text):
    recall = sum(k in summary for k in must_keep) / len(must_keep)
    ratio = count_tokens(summary) / count_tokens(source_text)
    unsupported = call_llm("List the claims in the SUMMARY that do not appear in the SOURCE",
                           summary=summary, source=source_text)
    return {"recall": recall, "compression": ratio, "unsupported": unsupported}

must_keep is a list you mark up yourself from real logs, standing in for the reference summary you do not have. For the example in step 4, that list must at least contain ORD-1042 and VD-88317.

The three numbers have to be read together: high recall with a poor compression ratio means you have not compressed anything, while good compression with a long unsupported list means the summary is making things up.

The mistakes that still bring long sessions down

The first mistake is jumping straight to summarization when clearing old tool results would have been enough, which costs an extra model call and adds the risk of losing details. The next is cutting at a fixed number of messages without checking tool call/result pairs.

Another is re-summarizing from scratch every time, so details erode over successive rounds. The hardest to spot is placing the summary in the middle of the context. Because models make poor use of information in the middle of a long context, put the summary at the start and the most recent turns at the end.

How this skill shows up at the client

It usually arrives as a complaint: “the agent gets dumb when it runs for a long time”, “the cost per session is unusually high”. The first thing to do is ask for a log of a failing session, plot tokens turn by turn and find which tool is bloating the context.

When reading job descriptions for FDE or agent engineer roles, watch for phrases such as “long-running agents”, “context management” and “memory”. On a CV, “used LangChain” says little. State how much you cut tokens per session, in percent, what recall you held for the must-keep details and which test suite you measured it with.

An agent that runs for three hours and still remembers the customer ID from the first minute is a result the client feels immediately, even though they will never see the compact function behind it.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
5 sources
Read next on the roadmap · Stage 3: Applied AITeach your RAG agent to search again instead of bluffingA grade-and-rewrite loop catches near-miss passages, and 20 real questions with known answers show whether the loop actually helps.