Multi-turn RAG in practice: rewrite follow-up questions, then measure with your own test set

Picture a client’s internal chatbot receiving three questions in a row. “How many days of annual leave do probationary employees get?” “What about seasonal contracts?” “Does that carry over?” The first two get fluent answers. The third gets something unrelated.

The language model is not what breaks here. The fault lies one step earlier. A guide to LlamaIndex’s CondenseQuestionChatEngine describes exactly this situation: the retriever has no idea what “that” is. When “does that carry over?” is embedded verbatim, it can match any document that mentions carrying something over.

If you are an FDE, your demo usually has a single question, while the client’s real users ask in chains. This article walks through a process you can run on a laptop today: build a multi-turn test set, measure a baseline, add a rewriting step, then read the results in a way that will convince the client.

What does the benchmark say about later turns?

mtRAG is a human-generated conversational RAG benchmark of 110 conversations across 4 domains, averaging 7.7 turns each, for 842 tasks in total. Its authors draw two conclusions that should shape how you build your test set.

Every model struggles, most visibly on unanswerable questions and on later turns. And query-rewriting strategies consistently retrieve better than using only the last-turn question.

Rewriting does not solve everything, though. According to mtRAG, non-standalone questions remain hard even with rewriting’s help. The MTRAG-UN version released in 2026 focuses on exactly this difficult group: unanswerable questions, underspecified questions and non-standalone questions, with 666 tasks and more than 2,800 conversation turns across 6 domains.

Your test set therefore needs to measure three separate things: which turn it is, whether the question stands on its own, and whether the question has an answer.

Step 1: Build the test set before writing code

You need a working RAG pipeline (any vector store), a model you can call through an API, and Python. The code below is simplified: retrieve and call_llm are two functions you wire into your own system, not the API of any library.

Each line in the JSONL file is one question, together with the history up to that turn:

{"conv_id": "hr-01", "turn": 3,
 "history": [["user", "How many days of annual leave do probationary employees get?"], ["bot", "..."],
             ["user", "What about seasonal contracts?"], ["bot", "..."]],
 "question": "Does that carry over?",
 "type": "non_standalone",
 "gold_doc_ids": ["leave-seasonal-carryover"]}

The type field takes one of three values: standalone, non_standalone or unanswerable. For unanswerable questions, leave gold_doc_ids empty. Take the questions from the client’s real logs, anonymised, because real users ask far more tersely than anything you would invent.

Check: count the lines by type and by turn. If nearly all of them are turn 1 and standalone, the test set is measuring your demo, not the real product.

Step 2: Measure a baseline with only the last-turn question

import json
from collections import defaultdict

def hit_at_k(retrieved_ids, gold_ids):
    return any(d in gold_ids for d in retrieved_ids)

def evaluate(rows, make_query, k=5):
    buckets = defaultdict(lambda: [0, 0])  # hit, total
    for r in rows:
        if r["type"] == "unanswerable":
            continue  # measured separately in step 4
        q = make_query(r["history"], r["question"])
        ids = [d.id for d in retrieve(q, k=k)]
        key = (r["type"], "turn1" if r["turn"] == 1 else "turn2+")
        buckets[key][0] += hit_at_k(ids, r["gold_doc_ids"])
        buckets[key][1] += 1
    for key, (h, n) in sorted(buckets.items()):
        print(key, f"{h}/{n}")

rows = [json.loads(l) for l in open("multiturn_eval.jsonl")]
evaluate(rows, lambda history, q: q)  # baseline: last turn only

Check: print the results per group; do not collapse them into an average. Suppose the test set has 10 first turns and 20 follow-ups, and the baseline hits 9/10 on first turns and 8/20 on follow-ups. The average of 17/30 looks passable, but it hides the fact that the system misses more than half of the follow-up questions.

Step 3: Rewrite the question, but only when there is history

The goal of this step is to turn a follow-up into a self-contained query for retrieval. One design detail is worth keeping, and the documentation for the RAGLight library does the same: if the history is empty, make no extra model call.

REWRITE_PROMPT = """Using the conversation history, rewrite the last question
as a self-contained question, replacing every pronoun with the specific subject.
Do not answer the question. Return only the rewritten question.

History:
{history}

Last question: {question}"""

def rewrite_query(history, question):
    if not history:
        return question  # first turn: no extra model call
    text = "\n".join(f"{role}: {msg}" for role, msg in history[-6:])
    return call_llm(REWRITE_PROMPT.format(history=text, question=question))

evaluate(rows, rewrite_query)

For the leave example, the output you are hoping for is something like “Does unused annual leave for seasonal-contract employees carry over to the next year?”. Keeping only the 6 most recent messages is a simplification. Tune that number to the length of the client’s real conversations.

Check: do not look only at the score. Print the rewritten question next to the original for every non_standalone turn. If the model answers instead of rewriting, or slips in a detail that is not in the history, fix the prompt before you trust any number.

Step 4: Measure unanswerable questions separately

Better retrieval can have a side effect: the bot grows more confident answering questions the documents never mention. According to mtRAG, this is precisely the group where every model is weak. Run the full pipeline on the unanswerable rows and check whether the bot says clearly that it found no information.

REFUSAL_MARKERS = ["could not find", "no information"]  # simplified

unans = [r for r in rows if r["type"] == "unanswerable"]
ok = sum(any(m in answer(r).lower() for m in REFUSAL_MARKERS) for r in unans)
print(f"Correct refusals: {ok}/{len(unans)}")

String matching is crude. As the test set grows, you can replace it with a human grader or a model grader with a rubric. Either way, this metric must sit next to hit@k in every report you send the client.

Step 5: Work out the cost before enabling it for every user

The rewriting step is not free. The RAGLight documentation advises turning the option off if latency and model cost matter more than resolving references within a conversation.

There are two ways to bring the cost down: skip rewriting on the first turn, as the code above does, and use a cheaper model for this step. The LlamaIndex guide uses gpt-3.5-turbo for rewriting as its example.

The decision is fairly simple. Rerun Step 3 with the cheap model. If the number of follow-ups retrieved correctly barely changes, you have your answer. And if the logs show that the client’s users almost always ask one question and stop, the measured results are a sound reason not to enable rewriting.

Common mistakes and how they show up at the client

The most common mistake is a test set made up entirely of first turns. The second is leaving out unanswerable questions altogether, so rewriting looks like a pure improvement. The third is scoring without reading the rewritten questions, so by the time the client notices the bot inventing extra conditions, it is too late.

Rewriting is not the end of the road either. The SELF-multi-RAG study argues that conversational systems also need to decide for themselves when to retrieve, and its authors report response quality improving by about 13% according to human raters.

That figure measures answer quality, not recall, so do not compare it with your hit@k.

On the client side, this skill usually comes up as a vague complaint such as “the bot gets dumber the longer you talk to it”. You have an advantage if you can answer with a table broken down by turn and by question type.

When reading job descriptions, look for phrases such as “evaluation”, “conversational” and “retrieval quality”. On your CV, say that you built a multi-turn eval set and which question groups you improved; that is far stronger evidence than “worked on RAG”.

Next time a client’s chatbot starts answering off-topic, the first thing to do is open the logs and count how many questions start with “what about”, “that one” or “so then”.