Unicode Normalizer: Clean Up Hidden Characters with NFC, NFD, NFKC and NFKD

Two strings look identical on screen. Your equality check says they are different. Your database stores one but not the other. Your search returns nothing for a query the user swears they typed correctly. Welcome to the invisible world of Unicode — where "café" can be four characters or five, where a zero-width space can lurk inside an identifier, and where a Cyrillic "а" can impersonate a Latin "a" well enough to fool a human reviewer.

The Unicode Normalizer is a free, browser-based tool that solves this class of problems in seconds. Paste any text, choose one of the four standard normalization forms — NFC, NFD, NFKC, or NFKD — and instantly see the normalized output alongside the exact code point (U+XXXX) of every single character in both the input and the result.

Unlike a plain string converter, the tool is also an inspector. It reveals what is actually inside your text: combining marks that should have been precomposed, zero-width characters that parsers choke on, and lookalike glyphs that slip through visual code review. In this guide, we will cover why normalization matters, how the four forms differ, and the practical situations where this tool saves you an afternoon of debugging.

Why Use the Unicode Normalizer?

• Detect invisible differences. Combining marks, zero-width spaces (U+200B), and byte-order marks (U+FEFF) are invisible in every editor but still count as characters — breaking string equality, deduplication, and hashing.
• Standardize text storage with NFC. NFC is the recommended default for the web, most databases, and interchange formats. Normalizing on write means every future comparison just works.
• Map compatibility characters down to their canonical equivalents. NFKC and NFKD convert ligatures (fi → fi), superscripts (² → 2), full-width forms, and other compatibility glyphs into plain base characters.
• Debug search and matching bugs. If a search index built from raw user input, lookalike precomposed and decomposed sequences will never match each other until you normalize both sides.
• Sanitize text pasted from Word, PDFs, and design tools. Office documents love smart quotes, ligatures, and superscripts. Normalize with NFKC before storing to keep your data clean.
• Spot homoglyph attacks. Displaying code points makes it obvious when a Cyrillic or Greek lookalike has replaced a Latin letter in a URL, identifier, or commit author name.

Key Features

• The dual inspector is the killer detail: seeing U+0065 U+0301 (e + combining acute) in the input become U+00E9 (é) in the output turns an abstract spec into something you can verify with your own eyes.
• The changed indicator tells you immediately whether normalization altered your text at all — useful when you are auditing data rather than fixing it.
• Everything runs client-side in your browser. Your text never leaves the page, which matters when you are inspecting tokens, usernames, or production data.

How to Use

1. Open the Unicode Normalizer and paste your text into the input box — anything from a single identifier to a whole paragraph.
2. Pick a normalization form. Leave it on NFC unless you have a specific reason; switch to NFKC when you want ligatures and superscripts flattened.
3. Watch the output update instantly, and check the changed indicator to see whether normalization actually modified anything.
4. Inspect the code-point panels beneath both text areas to find the exact U+XXXX values — this is where zero-width spaces, combining marks, and homoglyphs reveal themselves.
5. Copy the normalized text, or copy the code-point list if you are documenting a bug report.

Understanding Unicode Normalization Forms

Unicode lets the same visual text be encoded in multiple ways. The letter "é" can be a single precomposed character (U+00E9) or the letter "e" (U+0065) followed by a combining acute accent (U+0301). Both render identically. Both compare as unequal. Normalization exists to collapse these equivalent sequences into a predictable shape.

NFC — Canonical Composition composes characters into their shortest precomposed form wherever one exists. "e" + combining acute becomes the single character é. NFC is the default recommendation for the web (HTML, URLs, most JSON APIs) because it produces the most compact representation and matches what keyboards typically produce.

NFD — Canonical Decomposition does the opposite: it splits every precomposed character into its base letter plus combining marks. É becomes E + acute. macOS and Linux filesystems (HFS+ and NFD-based normalization in general) historically work with decomposed forms, which is why filenames copied between systems can silently change representation.

NFKC — Compatibility Composition goes further than canonical equivalence. It maps compatibility characters — glyphs that exist for legacy or compatibility reasons — onto their canonical equivalents. The ligature fl becomes fl. The superscript ² becomes 2. The full-width Ａ becomes A. Then it composes the result like NFC.

NFKD — Compatibility Decomposition applies the same compatibility mapping, then leaves the result decomposed like NFD.

The practical rule: use NFC for storage and interchange, NFD when a system explicitly requires decomposed form, and NFKC/NFKD when you need to flatten fancy typography — think search inputs, filename sanitization, or anything fed into a regex that expects plain ASCII-ish text.

One caution worth internalizing: normalization is not transliteration. NFKC will not translate Greek omicron to Latin o — it only maps characters that Unicode itself declares compatibility-equivalent. For lookalikes outside those mappings (the classic Cyrillic "а" vs Latin "a" phishing trick), the code-point inspector is your detection tool: identical pixels, different U+XXXX values.

Practical Use Cases

Debugging "identical but unequal" strings

A login lookup fails for one specific user. Their email renders perfectly in the logs. Paste both the stored value and the incoming value into the normalizer and diff the code points: the incoming version carries a combining mark sequence (NFD) while the database row holds the precomposed form (NFC). Normalize both to NFC, and the lookup works. The same technique resolves duplicate-key violations, broken cache hits, and tests that pass locally but fail in CI.

Standardizing database and API input

Define normalization at the boundary: every string entering your system passes through NFC (or NFKC if you want aggressive flattening) before validation and storage. This gives you one canonical representation per logical value, which keeps unique constraints, foreign keys, and deduplication jobs honest. When onboarding legacy data — especially content exported from Word or PDFs — a normalization pass with NFKC removes ligatures, superscript digits, and full-width characters that would otherwise poison downstream matching.

Fixing search indexing

If documents are indexed in whatever form they arrived in, a query typed on one keyboard will not match a document authored on another. Normalize both indexed content and query strings to the same form (NFKC is a common choice for search because it aggressively unifies lookalike compatibility characters) and recall improves without any fuzzy-matching infrastructure.

Cleaning pasted text

Text copied from PDFs, Word documents, and design tools arrives stuffed with smart quotes, ligatures, non-breaking spaces, and the occasional zero-width character. Running it through the normalizer's inspector first tells you exactly what you are dealing with; then NFKC plus a quick manual check for U+200B and U+FEFF gives you clean text safe for Markdown, code, or configuration files.

Best Practices

• Normalize early, at the system boundary — the moment text enters your application, not later at each comparison site.
• Pick one form per system and document it. NFC for general interchange is the safest default; deviate only with a stated reason.
• Never normalize secrets or formatted content blindly. Passwords, tokens, and anything where exact bytes matter (signatures, hashes) should be treated as opaque.
• Test round-trips. If you normalize on write, verify that your display layer handles the normalized form correctly — especially for languages that rely on combining marks, like Thai, Vietnamese, and Hindi.
• Use the code-point inspector before assuming. The most common normalization mistake is guessing at the encoding; sixty seconds of inspecting actual U+XXXX values beats an hour of speculation.
• Watch for zero-width characters in user input. U+200B and U+FEFF survive most trim operations and silently break parsers, template matching, and profanity filters.

Start Normalizing

Next time a string refuses to equal its twin, do not squint at the screen — inspect it. The Unicode Normalizer shows you every code point, applies all four standard normalization forms, and runs entirely in your browser. No account, no upload, no data leaving your machine. Paste your problem string and watch the invisible become visible.

Related Tools You Might Like

• Unicode Escape Converter — convert text to \uXXXX escape sequences and back
• Unicode Lookup — search for any character and inspect its properties
• Whitespace Remover — strip extra whitespace once you have normalized the underlying characters

Happy normalizing!