The Bugs That Live in Plain Text

In March a user named José sent me a screenshot. The page greeted him as José, and he asked, with more patience than I deserved, why the page kept renaming him. I recognized the damage before I finished reading, because à followed by © is the fingerprint of one specific accident: a program wrote his name in UTF-8, and a second program read the bytes back as Latin-1.

The é in José is the code point U+00E9. UTF-8 stores it as two bytes, 0xC3 0xA9. Latin-1 assigns every byte its own character, so a decoder working under that assumption reports two characters where the writer meant one: 0xC3 maps to Ã, and 0xA9 maps to ©. Four correct bytes met one wrong assumption. When the accident runs twice, decode wrong, re-encode, decode wrong again, José becomes José. I have dug through old databases where a name survived four of these round trips and bloated into a full line of à and  confetti.

My bug lived in a contact form on an older service I still maintain. Every layer spoke UTF-8 except one MySQL connection that defaulted to latin1, and that single layer poisoned the whole pipeline. The fix took one line of configuration. The diagnosis ate an afternoon, and somewhere in the middle of it I decided to write down what plain text keeps doing to people who have shipped software for years.

Bytes mean nothing until you pick a decoder

A text file contains no characters. It contains bytes, and characters appear only after someone chooses a rule for reading them. The industry ignored this distinction for decades because everyone shared one rule. ASCII, standardized in 1963, assigned 128 codes to the English alphabet, the digits, punctuation, and a set of control characters, all inside seven bits. The eighth bit doubled the space to 256 slots, and every vendor and country filled positions 0x80 through 0xFF with a different alphabet. Western Europe settled on ISO 8859-1, better known as Latin-1. Russian systems used KOI8-R. Japan built Shift JIS. The byte 0xE9 meant é in Paris, И in Moscow, and the first half of a two-byte character in Tokyo, and no file carried a label saying which rule had produced it. Anyone opening a document from another country had to guess, and a wrong guess produced mojibake, the Japanese word for garbled text.

Unicode solved the naming half of the problem by assigning every character a number, called a code point, written as U+ and hex digits. É became U+00C9, the snowman became U+2603, and the catalog now holds more than 150,000 entries. A code point still is not a byte. You need an encoding to turn the numbers into a file, and the encoding that won is UTF-8, which Ken Thompson and Rob Pike designed in September 1992, sketching the bit layout on a diner placemat in New Jersey.

You can verify the reasons it won with a hex editor. Every ASCII byte from 0x00 through 0x7F encodes the character it always did, so every existing English-language file became valid UTF-8 retroactively, an installed base no committee could have negotiated. Characters beyond ASCII occupy two to four bytes, and each byte of those sequences sits at 0x80 or above, so no multibyte character can contain anything resembling a slash, a quote, or a NUL. Decades of C programs that scanned for path separators kept working without knowing the encoding existed. The lead byte of a sequence announces its length, continuation bytes carry a distinct bit pattern, and a parser dropped into the middle of a stream finds the next character boundary within three bytes, a property called self-synchronization. Fixed-width alternatives like UTF-32 offer none of this and quadruple the size of English text. UTF-8 now carries north of 98 percent of the web. One footnote from the old world persists: when a page declares Latin-1, every browser decodes it as Windows-1252, a Microsoft sibling that fills the Latin-1 control range with curly quotes and the euro sign, because enough pages lied about their encoding that the lie became the standard. The ƒ in José comes from that range.

Four answers to one question about length

My own first name survives every encoding accident, because Hemanth fits in ASCII. The moment a string leaves that range, the question of how long it is stops having one answer. The flag emoji 🇺🇸 makes a tidy specimen. Your eye sees one symbol, which Unicode calls a grapheme cluster. It contains two code points, the regional indicator symbols U+1F1FA and U+1F1F8. UTF-8 stores it in eight bytes. JavaScript reports its .length as 4, because JavaScript strings count UTF-16 code units, and each of those code points lives above U+FFFF, where one code point costs two units. The language committed to that representation back when 65,536 characters looked like enough for anyone.

The family emoji goes further. 👨‍👩‍👧‍👦 is four person emoji stitched together with three zero-width joiners, U+200D: one grapheme, seven code points, twenty-five UTF-8 bytes, and a JavaScript length of 11. Code that truncates input with slice(0, 140) will eventually cut through the middle of a surrogate pair and leave a lone surrogate behind, and whatever encoder later serializes that string replaces the orphan with U+FFFD, the black diamond question mark your users screenshot and mail to you.

MySQL complicates the count in its own way. Its encoding named utf8 stores at most three bytes per character, which covered the world until emoji arrived needing four; anyone inserting 😀 into a utf8 column gets an error or a string truncated at the emoji, depending on strictness settings, and the honest encoding hides under the name utf8mb4. When a count matters, pick the unit on purpose. Storage quotas want bytes, protocol limits usually want code points, humans reading a character counter want graphemes, and nothing wants UTF-16 code units, yet code units remain what .length hands you. JavaScript now ships Intl.Segmenter for walking graphemes, and iterating a string with for...of at least yields whole code points.

The invisible first column

The byte order mark produces bug reports where every visible line of code is correct. U+FEFF exists for UTF-16, where a decoder must learn whether bytes arrive big end or little end first. UTF-8 has no byte order, and the Unicode standard neither requires nor recommends a BOM there, but tools write one anyway as the three bytes 0xEF 0xBB 0xBF. Excel is the most prolific source. Its CSV UTF-8 export leads with those bytes because Excel itself relies on them at import time to recognize that the file is not in a legacy code page.

The damage follows a script I can recite from memory. You parse the CSV, split the header row on commas, and the first column name arrives with U+FEFF glued to its front. It prints as id, because U+FEFF has no glyph. It compares unequal to id, because the string holds an extra code point. Your import chokes on a column it can display but never match, or every value shifts one field left, and the logs look clean because the culprit is invisible in them. I lost an hour once to a Node service where JSON.parse rejected a config file that three different editors displayed as flawless JSON; Node does not strip a BOM when reading a file, the JSON grammar has no token for U+FEFF, and an old Windows Notepad had planted one at byte zero. A hex dump of the first sixteen bytes ended the hunt in seconds. My eyes kept insisting the file was clean, and xxd corrected them.

Two strings that print the same

José's name hides one more trap. Unicode spells é two ways: as the single code point U+00E9, or as a plain e followed by U+0301, the combining acute accent. Unicode calls the first form NFC, the composed normal form, and the second NFD, the decomposed one. Fonts render the two identically; equality operators refuse to. In JavaScript, a composed and a decomposed é fail ===, the composed side reports length 1, and the decomposed side reports 2. Console logging cannot rescue you here, since both sides print the same glyph. You end up staring at two log lines that match perfectly while the comparison between them fails.

Filenames are this bug's natural habitat. Apple's HFS+ filesystem stored names decomposed, so a file saved as José.pdf on a Mac leaves the machine carrying five code points in its stem, while a web form where someone types the same name produces four. Store the form's version in your database, receive the file's version in an upload, and the lookup misses even though every log line you add prints José.pdf twice. Git grew a macOS-specific setting, core.precomposeUnicode, because repositories containing accented filenames showed phantom untracked files whenever Mac and Linux machines shared them. Since the José afternoon I normalize with normalize('NFC') at every boundary where text enters a system, and inside that boundary I let myself stop thinking about it.

Uppercase depends on where you are

Every developer I know would say the uppercase of i is I, and the mapping tables agree until the locale is Turkish. Turkish treats dotted and dotless i as separate letters in both cases: i uppercases to İ, U+0130, a capital I wearing a dot, and the dotless ı, U+0131, uppercases to the plain I. On a machine with a Turkish locale, code that uppercases a string and compares it against a constant, the pattern sitting in a thousand protocol handlers, watches mail become MAİL and the comparison fail. Java programs broke this way for years, in production, for Turkish users only, at the exact line where the author felt safest.

German teaches the length version of the same lesson. The sharp s, ß, uppercases to SS, so straße becomes STRASSE and gains a character. Unicode added a capital ẞ at U+1E9E in 2008, and German orthography admitted it in 2017, but the default case mapping still produces SS, so any code assuming that case changes preserve length has been wrong since before Unicode existed. Case-insensitive comparison, the most innocent feature a login form offers, requires case folding tables and a decision about locale, which goes some way toward explaining why identifier systems from DNS to programming language keywords keep retreating to ASCII.

I wrote back to José with an apology and a corrected page. The one-line fix closed his ticket; understanding why 0xC3 0xA9 ever became two characters is what keeps it closed. Each bug in this post shares an anatomy. Text poses as the simplest data type we have, and it stays simple only while both ends of a channel agree on encoding, normalization, and locale without ever saying so. Write the agreement down at every boundary your data crosses. The bytes will not volunteer it.