Deploying a prompt is moving a label, and why rollback can take up to a minute

Picture this: you have just moved the production label back to an older prompt version to contain an incident. For up to a minute afterwards, some of your running instances are still answering customers with the faulty prompt, because Langfuse’s SDK caches prompts on the client with a default TTL of 60 seconds. The rollback is done on the dashboard, but not necessarily for the customer.

That gap points to a bigger issue. A prompt in production is not a piece of text you edit and save. It is an artifact with a full deployment lifecycle: versioning, testing, promotion, rollback, and propagation delay too.

If your job is tuning LLM systems on a client’s site, this is a discipline to have in place before the first incident, not after it. To make it concrete, follow a hypothetical case from start to finish.

Deploying means moving a pointer

Say you run a support-ticket classification agent for a client. The client reports that the agent is filing refund complaints under “information requests”. You edit the prompt and add a paragraph of guidance on financial complaints. In Langfuse, that creates a new version, and the latest label automatically points to it.

But latest is not production. When code fetches a prompt without specifying a label, Langfuse returns the version labelled production. Keeping the two pointers separate is deliberate: you can create as many experimental versions as you like without the client seeing any of them, until you choose to move production to the new one.

Langfuse’s technical documentation on managing hundreds of prompts sums the model up in one sentence: moving a label is the deploy, and rollback is moving it back. No build, no release, no code change. The only thing that moves is the pointer.

Once deploying is just moving a pointer, the next question is who is allowed to move it. Langfuse offers protected labels, depending on the plan: once a label is protected, the viewer and member roles cannot move or delete it. On the Enterprise plan, an audit log records promote and set-label actions.

In the case above, if the production label is protected and your colleagues on the client side hold only viewer or member roles, they cannot move the label themselves to push a “trial fix” into production.

The pointer moves only after evals pass

Back to your fix. Before moving the label, the question has to be: where is this version better than the old one, and does it break anything? Braintrust recommends running automated evals on every pull request that changes a prompt.

TianPan, writing on prompt versioning discipline, goes a step further: every PR that touches a prompt file triggers automated evals, and a failing eval blocks the merge.

Think of this as unit testing for prompts. And as with unit tests, the quality lies in the test cases. Anthropic’s documentation on building evals advises designing tests that closely match the real task distribution, without forgetting edge cases.

For the ticket classifier, the eval suite should contain complaints, information requests and technical faults in the same proportions the client actually receives, plus the hard cases: tickets written partly in English, tickets that ask a question and complain at the same time.

The “add guidance on financial complaints” fix might raise the complaint score while causing the agent to start classifying questions about shipping fees as complaints. Only evals on the real distribution will catch that before the client does.

Rollback is a rehearsed script, not improvisation

Suppose the evals pass and you move production. Two days later, the client reports a new bug that the eval suite did not cover. This is where TianPan’s point becomes concrete: rollback must be planned in advance, not improvised during the incident. Braintrust describes the same idea from the traffic side: reverting means sending traffic back to the last known good version.

“Planned in advance” means three things in this case. You know exactly which version is last known good, not “last week’s version seemed fine”. You know who has permission to move the label and what the operation involves: in Langfuse, reassigning the production label to the older version, with no code redeploy.

And you know the delay: with the default 60-second cache, you tell the client “within a minute”, not “immediately”.

Then go back to the testing step. The new bug the client reported must become a test case in the eval suite, so that next time the pointer cannot move if the bug reappears. That closes the loop, and each incident makes the eval suite thicker instead of just making you more tired.

How to tell the ticket story on your CV

If you are preparing for an FDE role, the ticket classifier case above is exactly how to tell it on your CV and in interviews.

Do not write “experienced in prompt engineering”; explain how you fixed the prompt for the misfiled refund complaints, which eval blocked the fix that broke shipping-fee tickets, who was allowed to move the production label, and how long the rollback took to take effect.

When reading a job description, phrases such as prompt management, eval pipeline or observability are your cue to bring out that story. A specific rollback case, even from a side project, carries more weight than any claim about “prompt optimisation”.

Clients do not see your versions. They see only where the production pointer is pointing, and how long it takes you to move it back somewhere safe.