# Hands-on Vietnamese–English RAG: measure wrong-language answers too

> Retrieving the right document is only half the problem, because an answer can still fail at the first thing the user notices: its language.

Original: https://fdetimes.net/en/guides/cross-language-rag-wrong-language-answers/

In XRAG, the cross-language RAG benchmark published by Amazon Science, every model tested struggled with one seemingly simple task when the question was not in English and the retrieved documents were: answering in the right language. A system can find the right document, understand its content and still reply to a Vietnamese user in English.

That is also a scenario FDEs meet often: a client's staff ask questions in their mother tongue, while the operations manuals and technical documents are in English. A combined evaluation score hides where things break, so you need a measurement set that shows which layer the failure sits in.

This guide builds a small pipeline and a set of 52 bilingual questions to measure each layer. The examples use Vietnamese, but the method applies to other language pairs.

## What do you need to prepare?

You need Python, access to an LLM, one of two embedding models (multilingual-e5-large or BGE-M3) and a set of English documents a few dozen pages long. You also need someone who reads Vietnamese to review the questions and answers.

The code below is simplified. `embed()`, `count_tokens()` and `retrieve()` are placeholder functions that you connect to whichever library you use. All figures for run results are illustrative.

## Step 1: Write the test before building the system

XRAG's test set has 5,200 samples: 4,000 monolingual (1,000 per language) and 1,200 multilingual (300 per language). Monolingual means the retrieved documents are only in English; multilingual means the corpus mixes English with the language of the question.

A client project does not need that scale. Keep the same 10:3 ratio and you have a starter set of 40 monolingual and 12 multilingual questions, small enough for a person to read every answer. Write the questions from the client's real documents and situations rather than copying them from the benchmark.

```json
{"id": "q017", "setting": "mono",
 "question_vi": "M\\u00e3 l\\u1ed7i E-4021 ngh\\u0129a l\\u00e0 g\\u00ec v\\u00e0 x\\u1eed l\\u00fd th\\u1ebf n\\u00e0o?",
 "question_en": "What does error code E-4021 mean and how do I fix it?",
 "gold": ["ops-manual#s12"],
 "must_contain": ["E-4021"]}
```

Every question has both versions because this pairing is the main diagnostic tool of the whole guide. Here `question_vi` holds the Vietnamese version, written as JSON Unicode escapes. The format is home-made and follows no standard.

Check: open the passage listed in `gold` and confirm it really can answer the question. Ask the Vietnamese reader whether the question sounds like the way staff actually talk or is just a word-for-word translation.

## Step 2: Chunk and add prefixes

The model card for multilingual-e5-large says it truncates text at 512 tokens. Imagine a 3,000-token section fed in whole: the last 2,488 tokens never reach the vector, and questions about that part will miss without any error. With a self-imposed limit of 400 tokens, this section needs at least 8 chunks.

The second trap is the prefix. The model card requires every input to start with `query: ` or `passage: `, including text that is not in English.

```python
# Simplified: count_tokens() and embed() are placeholder functions
def to_passage(chunk):
    text = "passage: " + chunk
    assert count_tokens(text) <= 512
    return text

def to_query(question):
    return "query: " + question

# Questions: embed(to_query(q)); documents: embed(to_passage(c))
```

Check: print a few inputs just before embedding and look for the correct prefix at the start of the string. This mistake does not stop the program; it only makes the results worse.

## Step 3: Measure both question versions on the same corpus

No LLM is needed at this step. All you need to measure is whether the top 20 contains the right passage, on the two question fields of the same test set.

```python
def hit_rate(rows, retrieve, field, k=20):
    hits = 0
    for r in rows:
        top = retrieve(r[field], k)  # list of passage ids
        hits += all(g in top for g in r["gold"])
    return hits / len(rows)

en = hit_rate(mono, retrieve, "question_en")
vi = hit_rate(mono, retrieve, "question_vi")
```

Suppose the English version hits 36/40 and the Vietnamese version 30/40. Going through each question, you find 4 that fail in both versions and 6 that fail only in Vietnamese.

These two groups need different fixes. The first 4 are probably a chunking or document problem, and changing the embedding model may not help. The other 6 are what is actually lost to language.

The e5 model card also warns that quality may drop for low-resource languages, so do not infer its ability in Vietnamese from one general benchmark score.

Check: turn off the prefixes and run again. If the Vietnamese version falls to 26/40, as in this illustrative example, you now have a number showing what the prefixes are worth on the client's data.

## Step 4: Let BM25 catch what needs no translation

Vietnamese questions often keep error codes and English product names unchanged. Those strings need no translation, so keyword matching can find them even when embeddings miss. Anthropic has also recorded that embeddings combined with BM25 do better than embeddings alone.

```python
# Rank-based merge; c=60 is only a starting value, not a required standard
def merge(dense_ids, bm25_ids, k=20, c=60):
    score = {}
    for ids in (dense_ids, bm25_ids):
        for rank, doc_id in enumerate(ids):
            score[doc_id] = score.get(doc_id, 0) + 1 / (c + rank)
    return sorted(score, key=score.get, reverse=True)[:k]
```

In the illustrative example, the Vietnamese version rises to 34/40, meaning failed retrievals fall from 10 to 6, or by 40%.

If you want to keep measuring with the same yardstick, Anthropic reports on the top 20 chunks that Contextual Retrieval cuts failed retrievals by 49%, and by 67% with reranking added. Those figures belong to a different technique, so treat them as a pointer to the next layer to test and do not turn them into a promise to the client.

Check: filter for the questions containing error codes or product names. If BM25 helps this group specifically, the result has a clear explanation; if not, check whether tokenisation is breaking strings such as `E-4021`.

## Step 5: Score content and language in two columns

Only now do you connect the LLM. You can start with an instruction in the prompt: the answer must be in Vietnamese and keep error codes and product names unchanged. But given XRAG's finding, that instruction is something to measure, not something to trust.

```python
import re, unicodedata

# Simplified: share of letters with Vietnamese diacritics; the 0.1 threshold is an arbitrary choice
# The character class lists those accented letters as Unicode escapes
VI = re.compile(
    r"[\\u0103\\u00e2\\u0111\\u00ea\\u00f4\\u01a1\\u01b0"
    r"\\u00e1\\u00e0\\u1ea3\\u00e3\\u1ea1\\u1ea5\\u1ea7\\u1ea9\\u1eab\\u1ead"
    r"\\u1eaf\\u1eb1\\u1eb3\\u1eb5\\u1eb7"
    r"\\u00e9\\u00e8\\u1ebb\\u1ebd\\u1eb9\\u1ebf\\u1ec1\\u1ec3\\u1ec5\\u1ec7"
    r"\\u00ed\\u00ec\\u1ec9\\u0129\\u1ecb"
    r"\\u00f3\\u00f2\\u1ecf\\u00f5\\u1ecd\\u1ed1\\u1ed3\\u1ed5\\u1ed7\\u1ed9"
    r"\\u1edb\\u1edd\\u1edf\\u1ee1\\u1ee3"
    r"\\u00fa\\u00f9\\u1ee7\\u0169\\u1ee5\\u1ee9\\u1eeb\\u1eed\\u1eef\\u1ef1"
    r"\\u00fd\\u1ef3\\u1ef7\\u1ef9\\u1ef5]", re.I)

def looks_vietnamese(answer):
    text = unicodedata.normalize("NFC", answer)
    letters = [c for c in text if c.isalpha()]
    return bool(letters) and len(VI.findall(text)) / len(letters) >= 0.1

def grade(row, answer):
    return {"id": row["id"],
            "content": all(s in answer for s in row["must_contain"]),
            "language": looks_vietnamese(answer)}
```

The `content` column uses `must_contain`: every required detail must appear verbatim, so this is the place for error codes, numbers and product names, the strings that stay the same across languages. Containing the string does not guarantee a correct answer, so a reviewer rereads the questions marked as passing and records the final result in this column.

The `language` column is only a rough filter, and an answer heavy with English terms can be failed unfairly. Read the answers close to the threshold by hand.

**Key point:** A correct answer in the wrong language is still a failed answer for the user.

Suppose that of 40 questions, 29 are correct in content but only 22 of those are in Vietnamese. A combined score cannot tell you this, while two separate columns show at once that the fix belongs in the prompt or the generating model, not the vector store.

If you score with Ragas, read the prompts before trusting the numbers. Ragas adapts metrics to other languages by using an LLM to translate the few-shot examples in its prompts, so one poorly translated example can skew every later scoring run. Once reviewed, save that exact version so that runs remain comparable.

## Step 6: Test a mixed corpus with the remaining 12 questions

Client documents are rarely clean in a single language; a corpus may also include internal notes in Vietnamese. The XRAG authors say the main difficulty of the multilingual setting is reasoning over information retrieved in several languages, so this set of questions must force the system to use both sources.

Imagine a question about a load limit: the figure is in the English manual, while the process for requesting an exception is in a Vietnamese note. `gold` contains both passages, and `must_contain` has one detail from each. Missing either half counts as a failure, even if the other half is entirely right.

`hit_rate()` and `grade()` already require every element, so they can be reused unchanged. Suppose both passages reach the top 20 on 10/12 questions, but only 6 of those get an answer covering both halves. The 2 questions missing a passage go back for a retrieval fix, while the 4 that had all the documents and still failed point to a reasoning failure.

Check: open the two `gold` passages of each question and confirm each can answer only one half. If one passage is enough to answer the whole question, that question does not force the system to use two sources and needs rewriting.

Report this group separately from the 40 monolingual questions, because the two settings fail in different ways: one in answer language, the other in cross-language reasoning.

## When should you switch to BGE-M3?

When long documents make the 512-token limit a real problem, BGE-M3 is a candidate worth trying. It also combines dense, sparse and multi-vector retrieval in one model, so it is worth comparing with running BM25 separately.

| | multilingual-e5-large | BGE-M3 |
|---|---|---|
| Languages | 100 | More than 100 |
| Maximum input | Truncates at 512 tokens | Up to 8192 tokens |
| Building inputs | `query: ` and `passage: ` required | No instruction needed on queries |
| What to do when switching to it | Add prefixes, rechunk, re-embed the corpus | Remove prefixes, reconsider chunk size, re-embed the corpus |

Do not change several things at once. Keep the 52 questions, swap the model and rerun all the steps so you know where any difference comes from.

## Mistakes that make the measurements meaningless

One easy mistake is using machine translation to turn English questions into Vietnamese for the test. Translated questions can be tidier than what real users type, so scores tend to look better than reality.

Another trap is tuning the prompt or chunking against the same 52 questions and then reporting scores on that set. Once the test has become tuning data, keep a few new questions aside for validation.

Nor should you swap models and forget to change the inputs or re-embed the corpus. The system still runs, but you are measuring something other than what you think.

## First job at the client site

Before writing any code, ask the support or operations team for 20 real questions, along with the documents they use to answer them. Also ask whether the corpus has Vietnamese versions, because the answer decides how large your multilingual set needs to be.

Then settle with the person in charge: what does "the right language" mean? Which terms must stay in English, and which must be translated? Write these agreements into the scoring criteria, so that nobody has to argue at the end of the project over whether an answer with mixed English terms passes or fails.

When applying for jobs, the thing worth showing is the measurement report, not the demo. In your CV, describe how you separated retrieval failures from language failures and state how many failures you eliminated, using real numbers from the project.

When reading job descriptions, lines mentioning evals, multilingual documents or deployment with client data are where this report belongs.

Next time a client asks "does the system understand Vietnamese?", the best answer is neither yes nor no, but three columns of measurements on their own documents.

**Try this week:**

- Take a public set of English product documents and write 10 bilingual questions containing error codes or product names.
- Run top-20 retrieval with and without the query/passage prefixes, then record the number of failures for each run.
- Write a one-page report with three columns for retrieval, content and language, plus a recommendation on which layer to fix first.

## Sources

- [AmazonScience/XRAG (Hugging Face dataset card)](https://huggingface.co/datasets/AmazonScience/XRAG)

- [XRAG: Cross-lingual Retrieval-Augmented Generation](https://arxiv.org/abs/2505.10089)

- [intfloat/multilingual-e5-large (Hugging Face model card)](https://huggingface.co/intfloat/multilingual-e5-large)

- [BAAI/bge-m3 (Hugging Face model card)](https://huggingface.co/BAAI/bge-m3)

- [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)

- [Adapting Metrics to Target Language (Ragas docs)](https://docs.ragas.io/en/stable/howtos/customizations/metrics/metrics_language_adaptation/)