Unicode / UTF-8 Inspector: The Hidden Machinery Inside Text

Text looks effortless because modern software hides an extraordinary amount of machinery. A letter arrives as a familiar shape, an accented word looks ordinary, and an emoji behaves as though it were one pictorial symbol. Beneath that surface, a computer may be handling Unicode code points, combining marks, UTF-16 code units, multi-byte UTF-8 sequences, invisible joiners, and normalization rules.

Consider the waving hand 👋. A reader sees one symbol. Unicode identifies one code point, U+1F44B; JavaScript stores it as two UTF-16 code units; UTF-8 encodes it as four bytes:

F0 9F 91 8B

Now consider the family emoji 👨‍👩‍👧‍👦. It also appears to be one symbol, but it is assembled from four person emoji connected by three invisible zero-width joiners. The result is one grapheme cluster made from seven code points, eleven UTF-16 code units, and twenty-five UTF-8 bytes.

Unicode identifies the elements of text. Encodings such as UTF-8 and UTF-16 determine how those elements are represented inside computers.

Even an ordinary é can be stored as one precomposed code point or as e followed by a combining acute accent. The two forms can look identical while containing different code points and bytes. A Unicode inspector reveals those layers instead of treating the word character as though it had only one meaning.

Unicode / UTF-8 Inspector at a Glance

The inspector follows two complementary routes:

visible text
  -> grapheme clusters
  -> Unicode code points
  -> UTF-16 code units
  -> UTF-8 bytes

raw bytes
  -> UTF-8 decoding
  -> Unicode text
  -> visible output

The first route asks what a string is made from. The second asks whether a sequence of bytes can be interpreted as valid UTF-8 text.

When One Character Is Not One Character

Early English-language computing encouraged a convenient assumption: one visible character was one byte. The letter A could be stored as 65, the next character occupied the next byte, and string length appeared to match the number of symbols on screen.

Unicode did not make human writing complicated. Human writing was already complicated. Unicode created a system broad enough to represent accents, thousands of script characters, emoji modifiers, flags built from regional indicators, and pictorial sequences joined by invisible formatting characters.

This is why character becomes unreliable in software discussions. It may mean a visible symbol, code point, UTF-16 unit, byte, or language-specific programming unit. Those meanings align for simple ASCII text and diverge almost everywhere else.

These differences affect how strings are measured, sliced, stored, searched, validated, and transmitted. A useful inspector therefore shows several counts instead of pretending that one number describes every layer.

Unicode: A Numerical Register for the World's Writing

Unicode is an international standard that assigns stable numerical identities to textual elements. Its scope extends beyond alphabetic letters to punctuation, mathematics, currency signs, combining marks, controls, emoji, historic scripts, and musical symbols.

A Unicode identity is called a code point and is conventionally written with U+ followed by a hexadecimal number.

U+0041 does not describe a font shape or a byte sequence. It identifies the abstract textual element A. A font determines how it looks; an encoding determines how it is stored.

Unicode's code space runs from U+0000 to U+10FFFF. Not every position is an assigned character. Some remain unassigned, some are reserved for private use, and U+D800 through U+DFFF are reserved for UTF-16 surrogate handling.

Before Unicode became dominant, systems frequently relied on incompatible encodings. The same byte could represent different characters under different conventions, so text transferred between systems could emerge as mojibake. Unicode supplied a shared character model, while UTF-8 and UTF-16 supplied practical representations of that model.

A code point is not automatically a byte. It is closer to an entry in an immense international register; an encoding decides how that entry is written inside a machine.

Grapheme Clusters: What People Perceive

Code points are fundamental to software, but they do not always correspond to what a reader experiences as one character. A grapheme cluster is a sequence of one or more code points that should normally behave as one user-perceived unit.

Consider a decomposed accented letter:

e + ◌́
U+0065 + U+0301

Rendered together, the sequence appears as é. A reader sees one accented letter. Unicode sees two code points. UTF-8 stores three bytes:

65 CC 81

Emoji make the distinction clearer:

This matters whenever software performs an operation intended to match human perception. Cursor movement, deletion, username limits, and truncation should normally respect grapheme boundaries. Cutting through a cluster can leave behind an isolated combining mark, modifier, or fragment of an emoji sequence.

Code points describe the elements. Grapheme clusters describe how those elements behave as visible units.

Code Points, Glyphs, and Fonts

A code point identifies a textual element; a glyph is the visual form drawn by a font and rendering system. The same code point can look different across typefaces, languages, platforms, and devices. Emoji designs vary considerably while preserving the same underlying code points.

The relationship also works in the other direction: several code points may contribute to one glyph. Combining accents attach to base letters, zero-width joiners assemble emoji sequences, and complex writing systems shape neighbouring letters according to context. A font may even render separate f and i characters as a ligature without changing the underlying text.

This separation is one of Unicode's strengths. Text remains text rather than becoming tied to one drawing. It also explains why inspecting the screen cannot reveal everything a string contains.

UTF-8: Converting Unicode into Bytes

Unicode assigns identities, but computers store and transmit bytes. UTF-8 is a variable-width encoding that turns Unicode scalar values into sequences of one to four bytes.

The surrogate range is excluded because surrogate values are UTF-16 storage components rather than Unicode scalar values.

UTF-8's first 128 values match ASCII, so ordinary English letters, digits, basic punctuation, and controls retain their familiar single-byte values.

A four-byte emoji is not visually larger than a one-byte letter. The width reflects where its code point falls in UTF-8's encoding design. Ten ASCII letters normally occupy ten bytes; ten supplementary-plane emoji normally occupy at least forty, before modifiers or joiners are counted.

How UTF-8 Reveals Its Structure

Leading bits indicate each UTF-8 byte's role:

The encoding of é is:

11000011 10101001

The first byte begins with 110, announcing a two-byte sequence. The second begins with 10, identifying it as a continuation byte.

The encoding of 👋 starts with 11110, so three continuation bytes must follow:

11110000 10011111 10010001 10001011

This self-describing structure helps decoders locate boundaries and reject malformed data. A continuation byte cannot legitimately begin a character, and a start byte is incomplete when its expected continuation bytes are missing.

UTF-16 and JavaScript's Unusual Length

JavaScript strings are organised around UTF-16 code units. A code unit is a 16-bit number, and many common code points fit in one:

A  -> 0041
é  -> 00E9
中 -> 4E2D

Code points above U+FFFF require a surrogate pair. For the waving hand:

👋 -> U+1F44B -> D83D DC4B

Those hexadecimal values are not separate characters. Together they represent one code point. This creates a famous JavaScript surprise:

"👋".length;          // 2 UTF-16 code units
Array.from("👋").length; // 1 code point

The difference grows with joined emoji:

Different operations expose different layers:

None is inherently wrong. Problems arise when code uses one measure while describing it as another.

Surrogate Pairs and Broken UTF-16

UTF-16 reserves high surrogates from D800 to DBFF and low surrogates from DC00 to DFFF. A valid supplementary character uses one high surrogate followed by one low surrogate.

An unpaired surrogate occurs when either half appears without its partner. Such malformed values can exist inside a JavaScript string even though they are not valid Unicode scalar values and cannot be directly represented in valid UTF-8.

TextEncoder replaces an unpaired surrogate with the replacement character U+FFFD, usually displayed as �. The resulting UTF-8 bytes therefore do not exactly represent the malformed UTF-16 units in the original string.

A programming language's internal string representation is not always identical to valid Unicode text. UTF-16 units are storage components; Unicode scalar values are the legitimate abstract values represented by UTF-8.

One é, Two Byte Sequences

The precomposed é uses one code point and two UTF-8 bytes:

U+00E9 -> C3 A9

The decomposed form uses an ordinary e followed by a combining acute accent:

U+0065 U+0301 -> 65 CC 81

To a reader, they may be the same letter. To byte-by-byte comparison, they are different strings. That can cause failed searches, duplicate-looking usernames, inconsistent filenames, cache misses, database mismatches, and different cryptographic hashes.

NFC, NFD, NFKC, and NFKD

Unicode normalization provides standard ways to transform equivalent or related strings into consistent forms.

NFC and NFD preserve canonical equivalence. NFKC and NFKD are more aggressive: they may collapse full-width letters, ligatures, superscripts, circled characters, and other compatibility forms.

Normalization is therefore a policy decision, not a universal cleaning step. Search indexing may reasonably collapse compatibility variants. A scholarly archive, protocol, or system preserving exact user input may not. The inspector compares all four forms without silently altering the original value.

Invisible Characters That Change Everything

Some of the most troublesome characters occupy no visible width, resemble ordinary spacing, or influence neighbouring characters without drawing a symbol of their own.

These characters are legitimate and often useful. Trouble begins when they enter unexpectedly. Two labels may look identical while containing different spaces. A search can fail because a zero-width character divides a word. Copied source code can contain invisible formatting marks.

Visualising whitespace and format characters turns those mysteries into inspectable data.

Scripts, Categories, and Lookalike Letters

Unicode records properties for code points. The general category describes the broad kind of character:

The script property identifies writing systems such as Latin, Greek, Cyrillic, Arabic, Hebrew, Han, Hiragana, or Devanagari. Shared punctuation and symbols often belong to Common; combining marks may be Inherited because their context comes from a neighbouring character.

This becomes important when text contains lookalikes from different scripts. A Latin a and Cyrillic а can appear almost identical in some fonts while having different code points. Such substitutions may be accidental, but they can also support deceptive usernames, domains, or source code.

Unicode is not at fault for representing similar-looking letters used by real writing systems. The security risk appears when software or people assume visual similarity guarantees textual identity.

Worked Inspection: Hello, café 👋

Most of Hello, café 👋 lies in ASCII. The é needs two UTF-8 bytes and the emoji needs four.

The complete byte sequence is:

48 65 6C 6C 6F 2C 20 63 61 66 C3 A9 20 F0 9F 91 8B

The string contains thirteen grapheme clusters and thirteen code points, but fourteen UTF-16 units and seventeen UTF-8 bytes. Nothing in its appearance announces those differences.

Decoding Bytes Back into Text

Inspection begins with text and reveals its encoded forms. Decoding begins with raw bytes and asks what they represent under UTF-8.

48 65 6C 6C 6F -> Hello

The same five bytes can be presented several ways:

These are not different messages. They are textual presentations of the same byte sequence. Decoding has two stages: parse the notation into bytes, then interpret those bytes as UTF-8.

Bad Byte Notation Is Not Bad UTF-8

A good decoder identifies the layer at which failure occurred.

FF is a legitimate raw byte value; it simply cannot occur in a valid UTF-8 sequence. C3 is also a valid byte, but it announces a two-byte sequence and needs a continuation. The representation parser and the UTF-8 decoder answer different questions.

Strict Decoding and U+FFFD

A strict, or fatal, decoder rejects malformed UTF-8. A tolerant decoder continues and inserts the replacement character U+FFFD, displayed as �.

For example:

48 65 FF 6C 6F -> He�lo

The replacement symbol does not reveal the lost character. It only records that valid text could not be recovered at that position. Tolerant decoding is useful when damaged text is better than no display; strict decoding is useful when repair would hide an important defect.

The UTF-8 Round Trip

A round trip tests whether decoding preserved the original bytes:

original bytes
  -> decode as UTF-8
  -> recovered text
  -> encode as UTF-8
  -> new bytes

Valid UTF-8 should normally return the same sequence. Repaired input will not. A tolerant decoder turns FF into U+FFFD, whose UTF-8 representation is EF BF BD:

FF -> � -> EF BF BD

The visible result exists, but the original data was repaired rather than recovered exactly.

Why String Length Is a Design Decision

Imagine an application limiting a username to twenty “characters.” The right unit depends on the purpose.

A storage layer and user interface may legitimately enforce different limits. The important step is choosing deliberately. Many text bugs begin when a program counts one layer and labels it as another.

Where Unicode Inspection Matters

Unicode problems rarely announce themselves as Unicode problems. They appear as a search that inexplicably fails, a name that cannot be found, an emoji cut in half, a copied password that does not work, or a file beginning with an invisible character.

• Web development: inspect form input before it passes through JSON, databases, or HTML.
• Search and indexing: compare normalization forms and expose mixed-script lookalikes.
• Database work: explain duplicate-looking records, failed joins, and byte limits.
• Files and protocols: reveal byte-order marks, line endings, and malformed sequences.
• Internationalisation: test scripts, combining marks, and variable-width encodings beyond ASCII.
• Security analysis: expose invisible formatting characters and deceptive code points.

The educational value is just as important. Watching one string expand into graphemes, code points, code units, bytes, and escapes makes Unicode's architecture concrete.

Unicode Is Not Encryption

Unicode, UTF-8, UTF-16, hexadecimal, binary, Base64, and JavaScript escapes are representations. They provide no secrecy.

SECRET -> 53 45 43 52 45 54
SECRET -> U0VDUkVU

No secret key was introduced, so the information is easily recovered. Encoding answers how textual identities become data. Encryption answers how information can be made unreadable without the appropriate key.

Encoding makes data interpretable. Encryption makes it confidential.

A Simple JavaScript Unicode Inspector

Modern browsers provide much of the machinery required to inspect text. TextEncoder produces UTF-8 bytes, codePointAt() reads complete code points, and charCodeAt() exposes UTF-16 units.

const encoder = new TextEncoder();

function toHex(value, width = 2) {
    return value
        .toString(16)
        .toUpperCase()
        .padStart(width, "0");
}

function formatCodePoint(codePoint) {
    return `U+${toHex(codePoint, codePoint <= 0xFFFF ? 4 : 5)}`;
}

function inspectText(text) {
    const records = [];

    for (const character of text) {
        const codePoint = character.codePointAt(0);
        const utf8Bytes = Array.from(encoder.encode(character));
        const utf16Units = [];

        for (let index = 0; index < character.length; index++) {
            utf16Units.push(character.charCodeAt(index));
        }

        records.push({
            character,
            codePoint: formatCodePoint(codePoint),
            decimal: codePoint,
            utf16: utf16Units
                .map(unit => `0x${toHex(unit, 4)}`)
                .join(" "),
            utf8: utf8Bytes
                .map(byte => toHex(byte, 2))
                .join(" ")
        });
    }

    return records;
}

console.table(inspectText("Aé👋"));

The for...of loop advances by code points rather than individual UTF-16 units. The emoji is yielded once even though its JavaScript .length is 2. codePointAt(0) reads the complete scalar value, while charCodeAt() reveals the surrogate pair underneath.

Counting Graphemes in JavaScript

Where supported, Intl.Segmenter approximates user-perceived boundaries:

function countGraphemes(text) {
    const segmenter = new Intl.Segmenter(
        undefined,
        { granularity: "grapheme" }
    );

    return Array.from(segmenter.segment(text)).length;
}

console.log(countGraphemes("A"));
console.log(countGraphemes("é"));
console.log(countGraphemes("e\u0301"));
console.log(countGraphemes("👨‍👩‍👧‍👦"));

Each example normally produces 1 even though its code-point and UTF-16 lengths differ.

Decoding UTF-8 in JavaScript

TextDecoder performs the reverse operation:

const bytes = Uint8Array.from([
    0x48, 0x65, 0x6C, 0x6C, 0x6F,
    0x2C, 0x20,
    0xF0, 0x9F, 0x91, 0x8B
]);

const decoder = new TextDecoder("utf-8");
console.log(decoder.decode(bytes)); // Hello, 👋

const strictDecoder = new TextDecoder(
    "utf-8",
    { fatal: true }
);

The ordinary decoder preserves as much displayable text as possible. The fatal decoder insists that the supplied bytes form valid UTF-8.

Common Questions

Is Unicode the same as UTF-8?

No. Unicode assigns code points and character properties. UTF-8 is one way to encode Unicode scalar values as bytes.

Is one character always one byte?

No. UTF-8 uses one to four bytes per scalar value, and one visible grapheme may contain several code points.

Why does JavaScript say an emoji has length 2?

JavaScript's .length counts UTF-16 code units. Many emoji require a surrogate pair and therefore occupy two units.

Why can identical-looking strings contain different bytes?

They may use different normalization forms, combining sequences, spaces, punctuation marks, scripts, or invisible formatting characters.

What does U+FFFD mean?

U+FFFD is the replacement character. It marks text data that could not be decoded or represented exactly.

Is Base64 a Unicode encoding?

Base64 represents arbitrary bytes as printable ASCII. Those bytes may contain UTF-8, but Base64 itself does not assign Unicode characters.

Why are UTF-8 bytes shown in hexadecimal?

Hexadecimal is compact and maps neatly onto bytes: every value from 0 to 255 fits into exactly two digits from 00 to FF.

The Real Lesson of Unicode

Digital text is a layered system in which human perception, abstract identity, programming-language storage, byte encoding, and visual rendering overlap without always agreeing.

For simple ASCII text, the layers often align:

one visible letter
= one code point
= one UTF-16 unit
= one UTF-8 byte

That tidy relationship is the exception, not the rule. An accent may be a combining mark. An emoji may require a surrogate pair. A family may be assembled from seven code points and invisible joiners. Two words can look identical while containing different bytes. Malformed UTF-16 may be repaired during encoding, while malformed UTF-8 can be rejected or displayed with replacement characters.

Unicode does not complicate text for its own sake. It provides the framework needed to represent the writing, symbols, and digital expression of the world without reducing them to the assumptions of one alphabet.

The Unicode / UTF-8 Inspector makes that framework visible: the grapheme a person perceives, the code points Unicode identifies, the UTF-16 units JavaScript counts, and the UTF-8 bytes machines exchange. What appears on screen is only the final surface. The inspector reveals the machinery beneath it.