Why Two Identical-Looking Strings Compare Unequal

A few weeks ago a teammate pinged me about a "ghost" bug. A user typed a name into the search box, got zero results, then opened the record in the admin panel and pointed at it on screen. The name was right there. The string in the search query and the string in the database looked pixel-for-pixel the same. They still compared unequal.

This is the most common Unicode trap I see, and once you have hit it once you start seeing it everywhere: in search that misses accented names, in dedupe that leaves duplicates, in filenames that refuse to match across operating systems. The fix is Unicode normalization, and it is worth understanding rather than copy-pasting.

One letter, two storage shapes

Take the letter é. On screen it is a single glyph. Inside a string it can be stored two completely different ways:

As one code point, U+00E9 (LATIN SMALL LETTER E WITH ACUTE).
As two code points, a base e (U+0065) followed by a combining acute accent (U+0301).

Both render identically. Both mean "é". But a byte-by-byte string comparison sees one code point on one side and two on the other, so it returns false. In JavaScript you can prove it to yourself in one line:

"é" === "é" // false — one is precomposed, one is base + combining mark

The reason this duality exists is historical. Unicode kept precomposed characters like é around so it could round-trip cleanly with older single-byte charsets, but it also lets you build the same glyph compositionally from a base letter plus combining marks. Both are legal. Both are common in real data. macOS keyboards and filesystems lean toward the decomposed form; Windows and the web lean toward the precomposed form. Mix data from both and you get silent mismatches.

The four normalization forms

Unicode defines four normalization forms, and the names look more intimidating than the idea behind them. There are two independent choices: compose versus decompose, and canonical versus compatibility.

NFC composes. It packs a base letter and its accent into one code point wherever a single precomposed character exists. This is the web default and the right choice for storage and comparison.
NFD decomposes. It splits each precomposed character back into its base plus combining marks. Handy when you want to strip accents (decompose, then drop the combining marks) or sort.
NFKC is NFC plus compatibility folding. On top of composing, it flattens formatting-only variants to their plain form.
NFKD is NFD plus that same compatibility folding.

Concretely: NFC composes, NFD decomposes. The "K" forms go further and fold away cosmetic distinctions. Fullwidth ＡＢＣ collapses to plain ABC. The circled ① becomes the digit 1. The ﬁ ligature becomes the two letters f and i. Superscripts drop to their base. This folding is lossy on purpose: you trade exact appearance for a single searchable shape, which is exactly why search engines and username-uniqueness checks normalize to NFKC before comparing. The flip side is that you should never apply NFKC to text you intend to display verbatim or store as the canonical original.

A worked example: unifying the two é

Open the Unicode Normalizer and paste a precomposed é. The tool reports a code-point count of 1 and a UTF-8 byte count of 2. Switch the form to NFD and watch the numbers change: the code-point count jumps from 1 to 2, the byte count from 2 to 3, while the glyph on screen does not move at all. The per-character view names the new arrival: a combining acute accent, U+0301, sitting right after the base e.

Now do the reverse. Paste the decomposed version (the e plus the combining mark) and switch to NFC. The two code points merge back into one, the count drops to 1, and you are holding U+00E9 again. That is the entire mechanism: NFC composes the pair into a single code point, NFD splits it back apart, and both directions are reversible because they are canonical. Run "José" typed on a Mac and "José" typed on Windows through NFC and they become byte-identical, so the equality check that used to fail now passes.

The byte-count delta the tool shows is the tell. If you normalize a column and the byte count drops, those rows were genuinely decomposed and your old comparisons were quietly missing them.

Where this actually bites: search, dedupe, filenames

Three workflows account for nearly every normalization bug I have debugged.

Search. If your index stores José as NFC and an incoming query arrives as NFD, a plain equality or prefix match never fires. Normalize both the indexed value and the query to the same form. Do it once, at write time and at query time, not scattered through every read path.

Dedupe. Merge a contact list from two sources and you will get Café and Café as separate rows because one is composed and the other is not. A naive dedupe keeps both. Normalize the key column to NFC first (or NFKC if fullwidth and ligature variants must also collapse) and the visually equal entries become byte-equal, so they fold into one. When the same column also carries width or ligature noise pasted from PDFs, NFKC is the form that flattens fullwidth ＡＢＣ, circled digits and ﬁ ﬂ ligatures in a single pass.

Filenames. This is where cross-platform teams get burned. A file named with a decomposed é created on macOS will not match the precomposed name your Windows or Linux service expects, and your sync tool reports a phantom rename. Pick one form, normalize on ingest, and the problem disappears.

One caveat worth stating plainly: CJK is not automatically safe. Chinese ideographs have no canonical decomposition, so 中文 is untouched by NFC and NFD. But NFKC still folds fullwidth Latin letters and half-width kana inside mixed text, so a string that looks entirely Chinese can still change under NFKC if it contains those variants. Check the byte-count delta before you assume nothing happened. Korean Hangul is the opposite case: 한 is one precomposed code point under NFC but decomposes into its jamo (ㅎ, ㅏ, ㄴ) under NFD, which matters for both search and font rendering.

The rule of thumb I follow

Normalize once, on input, to NFC for anything you store or display. Use NFKC only to build a separate search or uniqueness key, never as the stored original, because its folding is irreversible. And whenever a comparison "should obviously be equal" but is not, paste both sides into a normalizer and look at the code-point counts before you reach for anything cleverer.

If you want to go deeper into what each code point actually is, the character inspector breaks any string down code point by code point. To see the raw encoding behind a normalization change, text to hex shows the exact bytes shifting. And when you need to scrub a known variant across a whole document, find and replace text pairs naturally with a normalization pass.

Unicode normalization is one of those topics that feels academic right up until a search box swears your data does not exist. Spend ten minutes watching é compose and decompose, and the next ghost bug will take you ten seconds instead of an afternoon.

Made by Toolora · Updated 2026-06-13