# Why “Nguyễn Thị Huệ” doesn’t equal “Nguyễn Thị Huệ” in your join

> Unicode lets one Vietnamese letter be stored two ways, and without normalising first, the join drops customers without raising an error.

Original: https://fdetimes.net/en/guides/vietnamese-unicode-normalization-nfc-joins/

It is Friday and you are at the client’s office, joining a list of contracts against the customer table in their CRM. The results are missing exactly the people with names like “Nguyễn Thị Huệ”. You open both files and check every diacritic: they are identical. Yet `==` still returns `False`.

Most likely nobody made a typo. The usual culprit is Unicode normalisation: two renderings that look exactly alike can use different code points. Vietnamese is especially prone to this, because a single letter can carry both a vowel mark and a tone mark, and almost every sentence is full of accented letters.

For a forward deployed engineer this is not a minor technical detail. A broken join raises no error. It quietly drops part of the customer base, and the person who notices may well be the client rather than you.

## One “ệ”, two ways to store it

Unicode defines *canonical equivalence*: different sequences of code points can represent the same character. The letter “ệ” can be stored as a single precomposed code point, or as a base “e” followed by a dot below and a circumflex as combining marks.

Those two storage methods correspond to two normal forms. NFD is canonical decomposition: the Python documentation describes it as translating each character into its decomposed form. NFC decomposes first, then recomposes into precomposed characters.

Unicode’s UAX #15 specification states plainly that the purpose of these normal forms is to let software determine whether two Unicode strings are equivalent.

This is the crux. When comparing strings, the machine looks at bytes, not at how the letters appear. An explainer on Apple’s file systems describes exactly this phenomenon: two renderings that look identical but use different code points.

**Key point:** Two strings that look identical are not necessarily the same sequence of bytes.

## Where does NFD data come from?

You rarely know for certain which form each of a client’s systems produces. So it helps to remember a few sources that tend to slip NFD strings into data unnoticed.

A familiar one is file names exported from a Mac. The older macOS file system, HFS+, stores names in NFD: if you name a file in form C, the system converts it to form D. APFS, the newer file system, performs no normalisation at all, so two file names that look the same can carry different code points depending on which program created them.

Picture the scenario at the top of this article. The legal team sends you a folder of contract PDFs, each named after the customer and exported from a Mac. You take the customer names from the file names and join them against the CRM table, which was populated through a web form. If one side is NFD and the other NFC, every name with a diacritic is at risk of being missed.

## A runnable example: why the join misses

The Python snippet below reproduces that situation.

```python
import unicodedata as ud


crm = "Nguyễn Thị Huệ"                    # entered via a web form, NFC form

from_filename = ud.normalize("NFD", crm)  # simulates a name taken from a file exported on a Mac


print(crm == from_filename)                 # False

print(len(crm), len(from_filename))         # 14 19

print(ud.normalize("NFC", from_filename) == crm)  # True

```

(The comments read: “entered via web form, NFC” and “simulates a name taken from a file exported on a Mac”.)

Count the code points. In NFC, “Nguyễn Thị Huệ” has 14 characters. In NFD, “ễ” becomes three code points (e, circumflex, tilde), “ị” becomes two and “ệ” becomes three, so the string is 19 long. Those five extra code points are enough for every comparison, every hash key and every database index to treat these as two different people.

The length mismatch causes a second bug. The Vietnamese Unicode FAQ notes that only when “ệ” is in NFC will a length calculation always return 1; in other forms, languages such as Java or C# will return a different length.

A validation rule such as “name may be at most 50 characters” could reject a perfectly valid name simply because it came from a Mac.

## The fix: normalise once, at the entry point

The first step is to measure before you fix. Since Python 3.8, `unicodedata.is_normalized` tells you whether a string is already in the normal form you specify. It is the cheapest data-quality check you have.

```python
def audit(values, form="NFC"):
    mismatched = [v for v in values if not ud.is_normalized(form, v)]
    return len(mismatched), mismatched[:5]

def make_key(name: str) -> str:
    return ud.normalize("NFC", name).strip()
```

The second step is to pick one normal form and stick to it. For Vietnamese, choose NFC. W3C recommends that text transcoded from legacy encodings should end up in NFC, and the Vietnamese Unicode FAQ favours precomposed characters because they are easier to process and display and print better.

The third step is to normalise at the boundary, as soon as data enters the pipeline, and to apply it to both sides of the join. In pandas it is a single line: `df["key"] = df["customer_name"].map(make_key)`. The original column stays untouched so you can cross-check when the client asks.

The final step is to report the numbers. How much did the match count change before and after normalisation? A sentence such as “after NFC normalisation, N more contracts were matched to their owners” is the kind of evidence that earns a client’s trust in your pipeline.

## Common mistakes

The most common mistake is the quick fix of stripping diacritics. A join on unaccented strings will certainly match, but “Huệ” and “Huế”, or “Lan” and “Lân”, will collapse into one. You trade an easy-to-spot missing-data bug for a much harder-to-spot bug that assigns records to the wrong customer.

The second mistake is normalising only one side. You fix the file in your hands, but the table in the client’s database may still hold a few NFD rows from an old import.

W3C’s principle applies here: canonically equivalent strings must be treated as if they were identical, so comparisons must go through the same function everywhere.

The third mistake is switching to NFKC or NFKD because they seem “stronger”. W3C warns that compatibility decomposition loses meaning: the superscript “²” is folded into “2”, and the ligature “ﬁ” into “fi”. For Vietnamese customer-name keys, NFC is enough.

The last mistake is trusting your eyes. When two “identical” strings fail to match, stop staring at the screen; print `len()` or the list of code points. The habit will save you an entire afternoon.

## From a fixed join to a line on your CV

The four steps above also make good material for your CV.

Don’t write “proficient in Unicode”. Describe the actual process: measured with `is_normalized`, normalised to NFC at the entry point on both sides of the join, and raised the match rate from X to Y. A line like that shows a hiring manager that you find silent bugs before the client has to.

Next time a join is missing a few rows for no obvious reason, the first question to ask is not “who made the typo?” but “which normal form is this string in?”

**Try this week:**

- Pick a customer-name column in your current project, run unicodedata.is_normalized('NFC', ...) on every row and count the rows that are not normalised.
- Create a file named “Nguyễn Thị Huệ.txt” on a Mac, read the file name in Python, print len() and compare it with a hand-typed string.
- Write a make_key() function shared by every pipeline that joins Vietnamese data, and add a unit test with an NFD string.

## Sources

- [UAX #15: Unicode Normalization Forms (Unicode 18.0.0, revision 58)](https://unicode.org/reports/tr15/)

- [Character Model for the World Wide Web: String Matching (W3C First Public Working Draft)](https://www.w3.org/TR/charmod-norm/)

- [unicodedata — Unicode Database (Python 3.14.8 documentation)](https://docs.python.org/3/library/unicodedata.html)

- [Explainer: Unicode, normalization and APFS](https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/)

- [Vietnamese Unicode FAQs](https://vietunicode.sourceforge.net/faq.htm)
