Letta: building agents with persistent memory, and when not to give them to a client

With Letta, everything an agent has ever seen and done is written to a database: memory, user messages, reasoning and tool calls. That sounds like the cure for forgetful agents. But what the agent actually “remembers” when it answers is a different matter entirely.

Letta writes on its blog that, ultimately, memory is about choosing which tokens are placed in the context window at any given moment. That line should be every FDE’s starting point before promising a client an agent that “remembers users”.

If you are building a customer-support agent, an internal assistant or a copilot for an operations team, sooner or later the client will ask why the bot knew them yesterday but not today. Letta is a serious answer to that question. It comes at a price, though, and there are cases where you should not use it.

Where Letta came from, and what it does

Letta grew out of the MemGPT paper from a UC Berkeley group (Packer, Wooders, Lin, Fang, Patil, Stoica, Gonzalez), submitted on 12 October 2023. The idea was to manage a “virtual context” in the manner of an operating system’s memory hierarchy, because LLMs are constrained by their context window.

The open-source repository, licensed under Apache-2.0, describes itself as a platform for stateful agents, states plainly that it was formerly called MemGPT, and can be self-hosted with letta server.

Letta’s documentation defines a stateful agent as one that keeps memory and context across many conversations. To do this, Letta splits state into two layers that you must keep distinct.

The first layer is memory blocks. Blocks attached to an agent are always in context, pinned into the system prompt. Blocks can be edited through the API, so you, or the agent itself, can inspect and change what the agent is “remembering”. The second layer is everything else: history, reasoning, tool calls. These sit in the database and enter the context only when selected.

Every pinned token costs money on every call

Picture a technical-support assistant for B2B clients. You create a “client profile” block recording the service plan, the systems they run and past incidents. Thanks to this block, the agent does not have to ask again for things the user has already said.

Now do the arithmetic. Suppose the block is 2,000 tokens long and the agent receives 1,000 calls a day: the profile alone accounts for 2 million input tokens a day, even when the question is only “where do I change my password?”

Because the block is always pinned, it takes up room in context on every call. The design question is whether this information is needed on every turn, or only needs looking up when relevant.

Conversation history has its own price. Letta acknowledges that eviction and summarisation both lose information: older messages gradually lose their influence. Its advice is to evict only part of the history, for example 70%, to preserve the thread of the conversation.

Concretely, with 100 messages about to overflow the context, you push the oldest 70 into a summary and keep the 30 most recent intact. Any detail that existed only in those 70 messages may fade.

When should you tell the client “no”?

The first case is a task with no real state: classifying tickets, extracting fields from invoices, summarising a document. Calling the model directly with a fixed prompt is easier to debug, easier to write evals for, and cheaper. Adding a state database here only adds places for things to break.

The second case is sensitive data. Letta stores both user messages and reasoning in the database, so before switching it on you need to settle with the client’s security team where the database lives, who can read it, and how to delete data when a user asks.

Self-hosting with letta server keeps the data on the client’s infrastructure, but it does not replace that conversation.

The third case is a client that needs a system to stay stable for years. Letta has dropped the MemGPT-style loop in favour of “Letta V1” for the new generation of reasoning models: heartbeats and the send_message tool are deprecated, while the old architecture is still supported. Letta itself documents both the gains and the losses; for example, reasoning is often available only in summarised form and cannot be edited.

The SDK layer is changing too. The Letta V1 SDK is being deprecated to make way for the Letta Agent SDK, whose newest memory system is MemFS, whereas V1 supports only memory blocks. If you hand it over to a client, pin the version and write the upgrade path into the handover documentation.

Don’t buy a memory system on a benchmark number

Letta reports that its filesystem-only agent scored 74.0% on LoCoMo, against the 68.5% that Mem0 reported. This is a result Letta ran itself, so read it with caution.

Interestingly, in the same post Letta argues that benchmarks such as LoCoMo may be a weak signal, and that memory quality depends more on how the agent manages context than on any specific retrieval mechanism.

For FDEs, the lesson is to build your own eval on the client’s real conversations. Take 20 sessions in which users came back after a few days, write down what the agent must remember, then score whether it did. You will learn more than from any leaderboard.

What to learn first, and what to put on your CV

Learn the concepts before the tool: the context window, what gets pinned, what gets looked up, what gets evicted. Then run a self-hosted Letta and edit a memory block through the API to watch the agent’s answers change. API details will shift with the SDK; the habit of choosing tokens will still apply.

On your CV, don’t just write “used Letta”. Write that you designed memory for an agent, decided which information was pinned, measured the token cost, and where you declined to use persistent memory. When a job description mentions “agent memory” or “stateful agents”, the interview question most worth preparing is: when did you choose not to keep state?

An agent that remembers everything is not necessarily a good agent. A good FDE knows what the agent needs to forget.