Beyond Code & Karma · Encoding
One glyph is not one byte — and not always one code point. How a Unicode
code point becomes the 1 to 4 bytes a disk actually stores, why an emoji
is four bytes, why .length lies, and the law of
name and form hidden inside it.
Chapter 01 — The Foundation
Unicode assigns every character an abstract number — a
code point, written U+0041 for "A" or
U+1F600 for "😀". That number is an identity, not
storage. To put it on a disk or a wire you must encode it.
UTF-8 is the encoding that won the web: it turns one
code point into 1 to 4 bytes, and — crucially —
keeps every ASCII file byte-for-byte valid.
.length)
two UTF-16 code units. Conflate them and bugs breed.
0xxxxxxx byte. Every ASCII file is already valid
UTF-8 — the masterstroke that won the web.
The Cosmic Connection
In Vedic thought, all manifest things are
nama-rupa — name and
form. The nama is the eternal, formless identity; the
rupa is the body it wears to enter the material stream. A
character is exactly this: the code
point is its imperishable name, and the
bytes are the form it
takes to be stored and sent. The same syllable can wear 1 to 4
bytes — one meaning, many embodiments. The
akshara ॐ, the
"imperishable syllable", is itself a multi-byte being:
U+0950 → three bytes, E0 A4 90.
U+0950 is ॐ in every system, on every machine, forever."The name is one and imperishable; the forms are many and made of bytes. Read the name to know the meaning; read the bytes to know the cost."
Chapter 02 — The 1-to-4 Byte Ladder
UTF-8 is variable-length and self-synchronising. The leading byte's
high bits announce the length; every following byte starts with
10, so a decoder can always tell a leading byte from a
continuation. Bigger code point → taller ladder.
0xxxxxxx — code points U+0000…U+007F.
The 7 ASCII bits ride in one byte. No continuation, no overhead.
110xxxxx 10xxxxxx — up to U+07FF.
Accents, Greek, Cyrillic, Hebrew, Arabic. 11 payload bits.
1110xxxx 10xxxxxx 10xxxxxx — up to U+FFFF.
Most CJK, Devanagari (ॐ lives here), arrows, symbols. 16 payload bits.
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx — up to
U+10FFFF. Emoji and the astral
planes. 21 payload bits.
A single function maps a code point to its byte count — the same branch UTF-8 takes internally:
function utf8Len(cp) { if (cp < 0x80) return 1; // 0xxxxxxx if (cp < 0x800) return 2; // 110xxxxx 10xxxxxx if (cp < 0x10000) return 3; // 1110xxxx 10xxxxxx 10xxxxxx return 4; // 11110xxx + three 10xxxxxx }
10xxxxxx
tails. They never start with 0 or 11, so
they can't be mistaken for a leading byte — which is exactly what
makes UTF-8 self-synchronising: drop into the middle of a
stream and you can always find the next character boundary.
Take the character's Unicode number — e.g. ॐ is U+0950 = 2384. This is the nama, the name to be embodied.
2384 is above U+07FF but below U+FFFF, so it needs 3 bytes: a 1110xxxx leader and two 10xxxxxx continuations.
Spread the 16 payload bits of 0x0950 into the x slots, left to right. Result: E0 A4 90 — the rupa, the form on disk.
| Code point range | Bytes | Bit pattern | Example |
|---|---|---|---|
| U+0000 – U+007F | 1 | 0xxxxxxx | "A" → 41 |
| U+0080 – U+07FF | 2 | 110xxxxx 10xxxxxx | "é" → C3 A9 |
| U+0800 – U+FFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx | "ॐ" → E0 A4 90 |
| U+10000 – U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | "😀" → F0 9F 98 80 |
You rarely lay out bits by hand. TextEncoder does it,
in the browser and in Node:
// The native encoder — code point → the UTF-8 bytes on disk. const bytes = new TextEncoder().encode("ॐ"); // Uint8Array(3) [224, 164, 144] → 0xE0 0xA4 0x90 console.log(bytes.length); // 3 — one akshara, three bytes
Chapter 03 — Live Visualization
Type a character and watch it split into its code point and the
UTF-8 bytes — header bits gold, payload bits cyan. Try the default
ॐ, then an emoji.
Chapter 04 — Deep Dive
Three different things hide behind the word "character" — code points, UTF-8 bytes, and UTF-16 code units. Mixing them up is the single most common encoding bug.
[...str].
TextEncoder().encode(s).length.
.length
counts — and where the lie lives.
A 16-bit unit can't hold a code point above U+FFFF, so UTF-16
stores it as a surrogate pair — two units. That's
why an emoji's .length is 2:
"😀".length // 2 — UTF-16 code units (a surrogate pair) [..."😀"].length // 1 — real code points "😀".codePointAt(0) // 128512 → U+1F600 new TextEncoder().encode("😀"); // 4 bytes: F0 9F 98 80
[...str].length or
Array.from(str).length for code points,
Intl.Segmenter for user-perceived characters
(graphemes), and read .length only when you genuinely
need UTF-16 code units (buffer sizing, slicing).
ASCII uses only code points 0–127, which all live in 7 bits. UTF-8
encodes exactly that range as a single 0xxxxxxx byte —
the high bit always 0. So the bytes of an ASCII file and the bytes
of the same text in UTF-8 are identical. This backward
compatibility is the main reason UTF-8 displaced UTF-16 and the
older code pages on the web: existing tools kept working unchanged.
Leading bytes start with 0, 110,
1110 or 11110; continuation bytes always
start with 10. Those sets never overlap. So if you
land in the middle of a multi-byte sequence — a truncated chunk, a
random seek — you skip forward past the 10xxxxxx bytes
until you hit a leading byte, and you're realigned. UTF-16 has no
such guarantee inside a buffer of raw bytes.
A family emoji 👨👩👧👦 looks like one character but is a whole
ZWJ sequence: man + woman + girl + boy, each a
4-byte emoji, glued by an invisible Zero Width Joiner
(U+200D). Seven code points, twenty-five UTF-8 bytes,
rendered as one grapheme. .length sees the units,
[...str] sees the code points, and only
Intl.Segmenter sees the one.
Off-by-one and corruption bugs cluster wherever the three counts get confused:
| Scenario | The trap |
|---|---|
| Char-limit validation | .length rejects a 1-emoji bio as "2 chars". Use grapheme count. |
| Slicing strings | slice() can cut a surrogate pair in half → a lone surrogate, broken glyph. |
| DB column sizing | "VARCHAR(10)" might mean 10 bytes, not 10 characters — an emoji eats four. |
| Reading byte streams | A chunk boundary can split a multi-byte sequence; decode with a streaming decoder. |
Chapter 05 — Dharma & The Byte
Karma is the law that the formless takes form to act in the world, and the form it takes is exactly fitted to the act. A code point — the eternal nama — does nothing until it descends into bytes to be stored, sent, read. The descent is not loss; it is how meaning enters the material stream.
const name = "ॐ"; // nama — the eternal code point U+0950 const form = new TextEncoder() // rupa — the body it wears to exist .encode(name); // [224, 164, 144] — 3 bytes, one meaning
The same name wears the body it needs — one byte for the plainest ASCII incarnation, four for an emoji rising from the astral planes. No name is privileged; each takes only the form its meaning requires. Read the bytes back and the name returns, undiminished — the form passes, the name is imperishable.
"The body is the cost of acting in the world; the name is what survives the decoding." — encoding, interpreted for systems engineers 🙏
Chapter 06 — Test Yourself
Three questions. No peeking at the visualizer. Your karma is watching.
F0 9F 98 80 — an 11110xxx leader plus three 10xxxxxx continuation bytes. Its JS .length of 2 is UTF-16 code units, a different count entirely..length counts 16-bit code units. Code points above U+FFFF can't fit one unit, so they're stored as a surrogate pair — two units. Count real code points with [...str].length, which gives 1.10xxxxxx. Leading bytes start with 0, 110, 1110 or 11110 — never 10. Because the two sets never overlap, a decoder dropped mid-stream can always resync to the next character boundary. That's UTF-8 being self-synchronising.The Enlightenment
0xxxxxxx byte. Every ASCII file is already valid UTF-8.
10xxxxxx. That single rule makes UTF-8 self-synchronising.
.length is UTF-16 code units, not characters. An emoji reads 2. Spread the string for code points.
कर्म करो, फल की चिंता मत करो