Counting UTF-8 Bytes vs Characters Without Getting Burned

The first time a database insert failed on me with a "value too long" error, I was sure it was a bug. The string was 18 characters. The column allowed 20. I counted twice. The insert still failed. The string contained two emoji and a Chinese name, and the column was measured in bytes, not characters. That gap between what you see and what gets stored is the single most common encoding mistake I run into, and it is worth understanding once so you never lose an afternoon to it again.

A character is not a byte

A character is a human unit. A byte is a storage unit. They only line up when your text is plain ASCII, where every letter, digit and punctuation mark is exactly one byte. The moment you step outside that range, the two numbers drift apart, and they drift apart by predictable amounts.

In UTF-8, the encoding the modern web settled on, the rules are simple to memorize:

ASCII characters (A, 7, ?, a space) are 1 byte each.
Most Latin-accented letters and the Greek and Cyrillic alphabets are 2 bytes each.
CJK characters — Chinese, Japanese kanji, Korean hanja — are 3 bytes each.
Many emoji are 4 bytes, and a single emoji built from modifiers or joiners can be far more.

So "character count" and "byte count" answer two different questions. A character count tells you how much a reader perceives. A byte count tells you how much storage, bandwidth or buffer space the text actually consumes. When you size anything physical — a column, a packet, an SMS segment — you want the byte count.

A worked example that breaks the intuition

Take the string café 中😀. Count the characters a person sees: c, a, f, é, a space, 中, another space, 😀. That is 8 characters.

Now count the UTF-8 bytes:

c, a, f are ASCII: 1 byte each, so 3 bytes.
é is a Latin-accented letter: 2 bytes.
The space is ASCII: 1 byte.
中 is a CJK character: 3 bytes.
The second space is ASCII: 1 byte.
😀 (U+1F600) is an astral-plane emoji: 4 bytes, encoded as f0 9f 98 80.

That sums to 3 + 2 + 1 + 3 + 1 + 4 = 14 bytes for 8 characters. Same string, two answers, and a 75% difference between them. If you had a 10-byte field and trusted the character count, this value would look like it fits with room to spare. By bytes, it overflows by four.

Why database column limits count bytes

Databases store the encoded form on disk, so a length limit is usually a byte budget. A VARCHAR(255) in many configurations means 255 bytes, not 255 characters. PostgreSQL, the legacy MySQL utf8 charset, fixed-width protocol buffers and most C string fields all measure the encoded size.

The practical consequence: a 20-byte name field holds 20 ASCII letters, but only about 6 Chinese characters (3 bytes each) or 5 four-byte emoji. If your test data is all English, the field looks generous. The first real user named in Cyrillic or Japanese is the one who hits the wall — at insert time, in production, where it is most expensive to discover. The fix is to size by worst case: paste your most punishing example into a UTF-8 byte counter, read the byte total, and pick a column width that clears it.

This is also where the related question "how big is this on disk" comes up, and once you have a byte figure you often want it in kilobytes or megabytes. A quick data storage converter turns the raw byte count into the unit your storage quota is actually written in.

Where JavaScript quietly misleads you

There is a third number hiding behind all of this, and it trips up frontend code constantly. JavaScript's String.prototype.length does not return characters. It returns UTF-16 code units. Characters outside the Basic Multilingual Plane — every emoji above U+FFFF, plus less common scripts — are stored as a surrogate pair of two code units.

So '😀'.length is 2, not 1. A form field that validates "max 140 characters" by checking .length will reject a tweet-length message that a user is certain is well under the limit, because each astral character silently counts double. To count real characters in JavaScript you need [...str].length, which iterates by code point. To count what a person actually perceives as one symbol — including emoji built from joiners — you need grapheme segmentation, which is different again.

This is why I stopped trusting any single "length" number. UTF-16 length, code point count and UTF-8 byte size are three distinct measurements, and the bug is almost always in the gap between two of them. Seeing all three side by side is usually enough to diagnose the problem in seconds: a frontend .length of 8 against a backend byte count of 14 immediately points at multibyte characters or a surrogate pair, not at a mysterious server error.

Emoji are the worst offenders

A plain emoji is one code point and four bytes, which is bad enough. Composite emoji are worse. A flag, a skin-toned thumbs-up, or a family glyph is several code points stitched together with zero-width joiners and modifiers. The family emoji 👨‍👩‍👧 looks like one symbol but is three people plus two joiners — and it can run to 18 or more UTF-8 bytes.

So "one emoji" can mean one code point or several, and one glyph on screen can cost anywhere from 4 bytes to a couple of dozen. If your length validation treats every visible symbol as one unit, a single family emoji can blow past a byte limit that comfortably held a full sentence of English. When you need to see exactly how a glyph decomposes, a Unicode character inspector shows the individual code points so the byte total stops being a surprise.

A short checklist to avoid the trap

When you are deciding a limit, ask which number the limit is actually measured in:

Storage and database columns — bytes. Size by UTF-8 byte count, worst case.
Network packets and protocol buffers — bytes. Trim by bytes so you never cut a multibyte sequence in half and corrupt it.
SMS segments and many chat APIs — encoded size, effectively bytes. One emoji can push a message into a second billed segment.
Frontend .length checks — UTF-16 code units, which match neither characters nor bytes. Validate against the real constraint, not .length.

Get into the habit of counting bytes whenever a limit is physical, and counting characters only when the limit is about readability. The two numbers are both correct; they just answer different questions, and picking the wrong one is what fails the insert.

Made by Toolora · Updated 2026-06-13