Wrong AI answers still return HTTP 200, so helpdesks need a runbook
First-line support does not need to understand the model, only to ask what just changed, gather three pieces of evidence and feel free to escalate early.
In brief
- A wrong LLM answer still returns HTTP 200, so a helpdesk checklist built on status codes will not catch it.
- Tier one does only three things: asks what just changed, sorts the failure into one of four groups, and attaches the prompt version, retrieved context and stored output.
- Tier two stops the bleeding with pre-approved actions, including a rollback in under ten minutes; tier three finds the cause and writes it back into the runbook.
The last week of an engagement. The client’s first-line helpdesk is used to tickets about failed logins, blank pages and 500 errors. Then an employee sends a screenshot: the internal AI assistant has given the wrong number of annual leave days. The logs show no red lines, so the person on duty closes the ticket with the note “could not reproduce”.
Old helpdesk checklists do not catch this kind of incident. The post “AI Incident Response Playbooks” on the tianpan.co blog points out that a fabricated LLM answer still returns status 200, along with a JSON payload that is correctly structured down to the last field. Every alert based on status codes treats it as a successful request.
If you leave the client without a way to classify this kind of failure, the ticket will come back to your inbox. Worse, it may land with nobody at all. The FDE’s job before handover is to teach the first line to recognize an AI failure, put it in the right group and escalate it to the right person.
The helpdesk needs to classify, not to understand the model
A common mistake is trying to teach the helpdesk about embeddings or temperature. They do not need any of that. They need a three-tier runbook in which each tier has clearly defined authority.
Tier one is the helpdesk: it takes in the report, classifies it, gathers evidence and opens an incident. Tier two is the client’s operations team, the people allowed to carry out certain pre-approved containment steps. Tier three is the engineering team that owns the system, which may still be you under a support contract, and it is responsible for finding the root cause.
The tianpan.co post proposes dividing AI incidents into four groups: model, prompt, data and infrastructure. Each group has a different owner and a different fix. The first minutes of a response should therefore go to putting the incident in the right group, before anyone tries to fix it.
The failure taxonomy must come from the client’s own traces
The four groups are only a general frame. The helpdesk needs concrete examples from the system they actually use, and the only way to get them is to read real traces. Hamel Husain, in his Eval FAQ, calls error analysis the most important activity in evals, because it exposes failure modes that only your application and your data have.
The method has two steps. The first is open coding: read each trace and write free-form notes, without a predefined taxonomy; Husain likens it to journaling. The second is axial coding: group the notes into a failure taxonomy and count the failures in each group. Husain considers this the most important step.
Say you read the traces of an HR assistant and jot a free-form note on each one: “quotes the old policy even though a new one exists”, “right document but wrong passage”, “answer cut off in the last sentence”, “timeout when asked about the pay scale”, “switched to bullet points since yesterday”. While writing, you do not need to name any groups.
When you group them, the first two notes become “retrieved the wrong document or wrong passage”, which belongs to data. The cut-off and the timeout belong to infrastructure, while the format change suggests the prompt was just edited. If the count shows retrieval accounts for most failures, that is the line you write most carefully in the tier-one checklist.
You stop when new traces no longer reveal new failure types or change the existing groups. Husain calls this theoretical saturation. The result is a list of failures with counts attached, and you place each failure type in one of the four groups.
| Group | What the helpdesk sees | First question |
|---|---|---|
| Prompt | Tone or answer format changes suddenly for every user | Was the prompt just edited? |
| Data | Wrong on one specific topic, cites an outdated or wrong document | Did anyone just upload or delete documents? |
| Model | Quality drops across the board even though prompt and data are unchanged | Did the provider change the model version? |
| Infrastructure | Slow, timeouts, answers cut off midway | Did the network, quota or configuration just change? |
The middle column is the part you must fill in yourself with examples from the client’s traces. The table above is only a starting point.
One ticket through three tiers
Back to the leave-days ticket. An LLM debugging runbook published on dev.to frames the triage question as: what changed, and on whose side? That question should also be the first line of the tier-one checklist, before anyone thinks about the model.
Suppose the person on duty asks the HR team and learns they uploaded a new set of policies on Monday. The failure only happens with questions about leave; other topics are still answered correctly. So this is a data incident. The person on duty opens a ticket using this template:
[AI INCIDENT] HR assistant gives wrong number of leave days
Detected at: Wednesday, 9:40
Suspected group: Data
Recent changes: HR uploaded new policy set on Monday
Evidence:
- Prompt version running at the time: (from the dashboard)
- Retrieved context: (list of documents and passages)
- Stored model output: (verbatim, not retyped)
Reproducible: Yes, 3/3 times with the original question
Escalate to: Tier two
The three evidence lines come from the dev.to runbook: the prompt version in production, the retrieved context and the full stored output. Without these three, tier three has to guess. So before handover, make sure the system stores all three and that the helpdesk knows where to find them.
Tier two stops the bleeding first and looks for the cause later
The tianpan.co post is blunt: contain the damage before you know why it happened. The tools are rolling back the prompt, pinning the model version or using feature flags. These are the actions you should pre-approve so the client’s operations team can take them on its own, without waiting for anyone.
The dev.to runbook adds two details worth copying into the handover document. Rollback time should be under ten minutes. And there should be a kill-switch prompt, a reviewed version known to work correctly, to switch to when everything else fails.
For the leave-days ticket, tier two could use a feature flag to turn off the leave topic and show a message pointing users to the original policy page. Only then does tier three check whether the new documents were chunked incorrectly or the old ones were never removed from the index.
Early alarms must be permitted in writing
First-line staff are often afraid to open an incident before they can prove the cause. Google’s SRE Incident Response Workbook sets the opposite principle: declare incidents early and often. The workbook also advises telling on-call staff explicitly that they are allowed to delegate and escalate while handling an incident.
Write that into the runbook as a single line: escalating carries no blame, even when it turns out to be a false alarm. Without that line, the helpdesk will drift back to closing tickets as “could not reproduce”.
Finally, every tier-three incident must feed back into the runbook. Google’s SRE book says there is no better way to learn than to document what has broken. A new failure type goes into the table; a new symptom goes into the middle column.
How runbooks usually die
The most common way is a failure taxonomy written from imagination rather than from traces. The helpdesk meets examples that do not match reality and abandons the runbook. The second is listing evidence to attach when the system does not yet store it, so the person on duty has nothing to fill in.
The third is giving tier two no authority beyond “escalate”, with no pre-approved actions, so every incident waits for an engineer. The fourth is a one-off handover in which nobody owns updating the runbook after each incident.
Once the work is done, put something more specific on your CV than “wrote handover documentation”: you built a failure taxonomy from real traces and designed a three-tier runbook that lets the client team handle AI incidents on its own.
The test of the handover is the first “wrong answer” ticket after you leave: whether it is classified correctly and reaches the right person without anyone calling you.
Was this article useful?
Thanks for the feedback!
5 sources
- Q: Why is "error analysis" so important in AI evals, and how is it performed? – Hamel's Blog · 2025-06-27
- AI Incident Response Playbooks: Why Your On-Call Runbook Doesn't Work for LLMs · 2026-04-20
- Debugging an LLM Bug at 3 AM: The Runbook I Wish I'd Had · 2026-04-18
- Chapter 13 - Emergency Response (Site Reliability Engineering)
- Chapter 9 - Incident Response (Site Reliability Workbook)