RAG in practice: when two customer documents disagree, make the system cite both

The WikiContradict benchmark has 253 human-labelled samples, and on it every model tested struggled to write answers that correctly reflect a contradictory context. They stumbled most on implicit conflicts, where two passages differ but neither says so outright.

Outside the lab, exactly this kind of contradiction already sits in the customer’s document store. Two versions of a procedure that disagree, an old PDF nobody deleted, a wiki written by two teams: all are common.

The MAGIC benchmark shows that both open and closed models are poor at detecting conflicts between contexts, and a system that cannot spot a contradiction can hardly tell the user that another document says the opposite.

The five steps below target that gap: label the sources in the context, agree on how to handle each type of conflict, write a prompt that forces quotes before the answer, keep citations in a machine-checkable form, and score the system on the customer’s own test set.

What do you need?

You need a working RAG pipeline, meaning a retriever that returns a few passages per question, access to an LLM, and Python. You also need about 10 to 20 real questions that the customer’s staff often ask. The code in this article is simplified and not tied to any particular library, so you will have to wire it into your own pipeline.

The whole article uses one hypothetical example. The customer is a retail chain. The file return-policy-2025.pdf says customers can exchange goods within 30 days, while the Store FAQ page says 14 days. A member of staff asks: “How long do customers have to exchange an item?”

Step 1: Every passage must carry its name and date

The MAGIC benchmark also found that models often fail to pinpoint which source a contradiction lies in. If you paste bare text into the context, the model has even less way of saying “according to document A”. So the first job is to give each passage an id, a title and an update date.

def format_context(chunks):
    parts = []
    for i, c in enumerate(chunks, start=1):
        parts.append(
            f'<doc id="D{i}" title="{c["title"]}" updated="{c["updated"]}">\n'
            f'{c["text"]}\n</doc>'
        )
    return "\n\n".join(parts)

Check: print the context for the exchange question. You should see both documents, each with its own date. If you see only one, the problem is in the retriever, not yet in the prompt. If the updated field is empty, ask the customer where to get the date before going further.

Step 2: Each type of conflict needs its own kind of answer

The paper DRAGged into Conflicts sets out a taxonomy of knowledge conflicts in RAG, with the desired behaviour for each type. The table below takes three of them that you will need to agree with the customer:

Conflict type	Sign at the customer site	What the system should do
Outdated information	Two versions of the same document, different dates	Prefer the newer version, optionally mention the old one
False information	One source has content that is likely wrong or misleading	Drop the false source
Differing views or results	Two teams, two studies, two approaches that are both reasonable	State the disagreement between sources clearly, without picking a side

The exchange example shows why the model cannot be left to guess. If the FAQ is simply an old version nobody updated, this is an outdated conflict and the answer is 30 days. But if 14 days is a separate policy for electronics, the user needs to see both numbers.

Only the customer knows which document is authoritative. So before writing the prompt, sit down with the document owner and write down a few rules, such as “the policy PDF always beats the FAQ” or “the team A and team B wikis are both valid”.

In this article’s example, suppose the customer confirms the following rule: “The Store FAQ and the policy PDF are both in force; 14 days applies to electronics, 30 days to all other product categories.” Under that rule, the exchange question is classed as a disagreement, and the answer must cite both documents.

Step 3: Make the model quote verbatim before answering

Two pieces of evidence support this step. Anthropic tested it with Claude 2 and found that asking the model to extract the relevant passages before answering raised accuracy from 0.939 to 0.961, a 36% reduction in errors, at the cost of slightly higher latency.

The authors of DRAGged into Conflicts also found that prompting the model to reason explicitly about conflicts in the retrieved documents improves answers, although there is still plenty of room for improvement.

Combining the two techniques gives a sample prompt like this (simplified):

Below are the documents, each with an id, a title and an updated date.

{context}

Question: {question}

Follow this order exactly:
1. In <quotes>, copy verbatim every relevant sentence, with the id at the start of each.
2. In <conflict>, say whether the quotes contradict each other.
   If they do, assign one type: outdated | false | disagreement.
3. In <answer>, answer according to these rules:
   - outdated: use the newer document, and state that the older document says otherwise.
   - false: drop the false document, and explain why.
   - disagreement: present both sides, each with its id.
Customer authority rules: {customer_rules}

Check: rerun the exchange question with {customer_rules} containing the rule the customer confirmed in step 2. The <quotes> section must contain a sentence from D1 and a sentence from D2. If it has only one, the model picked a side at the quoting stage, and that is a failure you need to add to the test set.

Step 4: Citations must be machine-checkable

Letting the model write ids into the text is enough for a trial run, but hard to verify at scale. If you use Claude, you can use the Citations feature. According to Anthropic’s documentation, it returns the exact passages that support each claim, so you can verify the answer and show sources to the user.

When handling conflicts, its value is that each part of the answer is tied to a specific passage. The interface can show “30 days” with a link to the PDF and “14 days” with a link to the FAQ. The API call syntax may change, so read the current Citations page rather than copying from an old article.

Step 5: Score with the customer’s own conflict test set

CONFLICTS is the first benchmark with explicit conflict-type labels: 458 queries, each with retrieval results and expert-assigned labels. You do not need 458 questions. Just borrow its structure for the customer’s 20.

{
  "query": "How long do customers have to exchange an item?",
  "docs": ["D1", "D2"],
  "conflict_type": "disagreement",
  "must_cite": ["D1", "D2"]
}

The disagreement label here comes from the rule the customer confirmed in step 2, not from your own guess. If one day the customer says the FAQ is only an old version, change the label to outdated and update must_cite to match the new rule.

import re

def cites_all(answer: str, case: dict) -> bool:
    found = set(re.findall(r"\bD\d+\b", answer))
    return all(doc_id in found for doc_id in case["must_cite"])

score = sum(cites_all(run(c["query"]), c) for c in cases) / len(cases)

The run function is your pipeline. With this simplified version, watch two things.

First, do not replace the regex with a substring check such as "D1" in answer. That would let “D1” match “D10”, and the code above extracts each id before comparing precisely to avoid this.

Second, if you use the Citations API, source ids sit in the citation part of the response, not in the text. In that case, read the ids from the citations rather than scanning the answer.

Run the test set with the old prompt, then with the new one, and record both numbers. Do not expect a perfect score: MAGIC shows that both open and closed models detect conflicts poorly, especially when multi-step reasoning is needed.

So deliberately include a few questions in the test set where the contradiction only emerges when two passages are put together.

Mistakes that waste all the work above

The most common mistake happens before the model is even involved. Deduplication can merge two near-identical passages and accidentally delete the other side, while a top-k that is too small retrieves only one side. In either case, no prompt, however well written, will help.

The second mistake is treating every conflict as a debate. If every answer ends with “document A says this, document B says that”, users will get irritated, especially when the old version has clearly expired. The table in step 2 exists precisely to prevent that.

The third mistake is checking only whether the answer sounds reasonable. A fluent answer that cites only one source is still wrong for a disagreement case, and only a check function such as cites_all will catch it.

The list of contradictions is the first thing to bring to the customer

On a real deployment, you use this skill from the first week. Run the conflict test set and take the list of contradictory document pairs to the customer. That list is often more useful to them than the chatbot itself, because it tells them which documents need cleaning up.

When reading FDE or applied AI job descriptions, look for words such as evals, grounding and citations. On your CV, be specific about what you did: how many questions your conflict test set had, and the share of answers citing all sources before and after the prompt change.

In interviews, walk through the 30-day versus 14-day example, then explain why you did not let the model choose on its own.

Models will keep getting stronger, but customers will still have two documents that say different things. The person who earns trust is the one who makes that contradiction visible rather than hiding it.