When you type an ñ and a server in Tokyo receives it intact, a worldwide agreement is working in the background: the Unicode standard and its dominant encoding, UTF-8. It may look like magic, but it is pure byte engineering, and understanding it explains a large part of how the internet was built.
The earlier chaos: one encoding per country
In the 80s and 90s there was no single standard for representing text. ASCII reserved 7 bits (128 values) for English letters, symbols and control characters, which left out virtually every other language. Each region built its own solution: ISO-8859-1 for Spanish and French, Windows-1252 for Microsoft systems, Shift-JIS for Japanese, EUC-KR for Korean. The result was a puzzle: the byte 0xE9 could be an é, a Greek letter, or part of a kanji depending on which tables each side used.
That disorder caused the famous “garbage characters”: mojibake, unreadable text because the sender encoded with one table and the receiver decoded with another. Sending an email with accented characters between Spain and a server in the US was a lottery.
Unicode: assigning a number to every symbol on Earth
Unicode solved the problem in two separate layers. The first is a catalog: every character (letter, ideogram, symbol, emoji) is assigned a code point, a unique number. It is written in hexadecimal with the prefix U+: the Latin A is U+0041, the ñ is U+00F1, the Chinese ideogram «好» is U+597D. Today there are more than 149,000 assigned code points, organized into 161 blocks, and the maximum space reaches U+10FFFF: more than a million possible values.
The second layer is the encoding: how to turn those numbers into bytes that a computer can store and transmit. This is where UTF-8 comes in.
UTF-8: variable length, inherited compatibility
UTF-8 (from Unicode Transformation Format, 8 bits per unit) uses a variable-length scheme: a character takes between 1 and 4 bytes depending on its code point. The genius is that the first 128 values match ASCII exactly. That means any valid ASCII document is also a valid UTF-8 document, which allowed a transition without breaking all existing software.
The trick lies in the continuation bytes. The most significant bit of each byte indicates whether it starts a new character or continues a previous one:
- 1 byte (0–0x7F):
0xxxxxxx— plain ASCII. - 2 bytes (0x80–0x7FF):
110xxxxx 10xxxxxx— up to Greek, Cyrillic, Hebrew. - 3 bytes (0x800–0xFFFF):
1110xxxx 10xxxxxx 10xxxxxx— most languages, including Chinese and Japanese. - 4 bytes (0x10000–0x10FFFF):
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx— emojis and rare historical characters.
The 1110 (or 11110) header marks the start of a 3- or 4-byte sequence; every continuation byte starts with 10. This design makes UTF-8 self-synchronizing: if you skip a byte or a fragment gets corrupted, you can realign at the next character without losing the whole string.
Why UTF-8 won (and not UTF-16)
Microsoft and Java bet on UTF-16, which uses 2-byte units. It has advantages for predominantly CJK (Chinese-Japanese-Korean) text because it takes less space, but two serious problems: characters above U+FFFF (including emojis) require surrogate pairs of two units, and the 0x00 byte that appears in almost every character makes it incompatible with ASCII. The internet, which was built on ASCII, chose UTF-8.
The victory was total: since 2008 UTF-8 dominates the web, and today about 98% of websites use it, according to the W3Techs observatory. It became the de facto standard for HTML (declared in the meta charset), JSON, XML, protocols and all kinds of APIs.
A problem the encoding does not solve: normalization
Here comes the subtle trap. Many characters can be written two ways in Unicode: precomposed or decomposed. The ñ can be the single code point U+00F1, or the combination of an n (U+006E) followed by a combining tilde (U+0303). They are two different byte sequences representing the same letter.
To compare correctly (for example, in a search engine or a database) you must apply normalization: convert everything to one of the four forms (NFC, NFD, NFKC, NFKD). If you don’t, “mañana” written one way won’t match the other when searching, and two visually identical passwords could generate different hashes.
The extreme case: emojis and grapheme clusters
An emoji like 👨👩👧👦 (family) is not a single code point: it is five characters joined by a Zero Width Joiner (ZWJ, U+200D), an invisible connector. At the byte level, a “simple” emoji can take up dozens of bytes. To split text without breaking it, looking at code points is not enough: you need to understand grapheme clusters, the unit the user perceives as a single character. That is why counting “characters” in JavaScript with string.length gives misleading results with emojis, and specialized libraries (like the native Intl.Segmenter) exist to segment them correctly.
Conclusion
Unicode and UTF-8 are one of computing’s greatest collaborative achievements: an agreement that lets any computer on the planet exchange text in any language without ambiguity. Next time your ñ crosses the Atlantic intact, you now know who to thank: a few well-designed bytes.
