unicode-inspector

Reveal, name, and categorize the invisible or deceptive Unicode codepoints hiding inside a string, then optionally strip or normalize them. These are the core modules — scan.js (detect) and strip.js (clean) — both pure and dependency-free. A CLI and a web UI are built in later tasks.

FRAMING LAW

This project exists for developer debugging, security review, and phishing analysis — full stop. Its purpose is to make hidden Unicode visible and understandable, so a human can decide what to do about it. No copy, code, metadata, keyword, or doc anywhere in this project will ever be framed as a way to evade text-classification systems, strip provenance markers, or disguise the origin of generated prose. Those framings are out of scope by policy, forever; the only sanctioned use cases are the three named above.

We ship the honest gap: an inspector that tells you the truth about a string, rather than a page selling ways to defeat someone else's checker.

Who it's for

• Developers debugging "why won't this string match / compare equal / parse?" — a stray U+200B (zero-width space) or U+00A0 (no-break space) is a classic invisible culprit.
• Security reviewers auditing source code and commits for Trojan-Source attacks (CVE-2021-42574), where bidirectional-control codepoints reorder how code displays versus how it compiles.
• Phishing / abuse analysts inspecting suspicious display names, URLs, and messages for hidden formatting characters.

What it detects

scan.js flags codepoints from a hand-curated, pinned table in codepoints.js, each carrying its official Unicode name and a class:

| class | examples | why it matters | | --- | --- | --- | | zero-width | U+200B ZERO WIDTH SPACE, U+200C/D, U+2060 WORD JOINER, U+FEFF (BOM) | renders to nothing; silently breaks equality, search, and tokenization | | bidi-control | U+202A–202E, U+2066–2069 | reorders visible text — the Trojan-Source code-review risk | | nbsp | U+00A0, U+2007, U+202F | space look-alikes that are not U+0020 | | soft-hyphen | U+00AD | invisible unless the line happens to break there | | variation-selector | U+FE00–FE0F | invisible selectors that mutate the previous glyph | | interlinear-annotation | U+FFF9–FFFB | invisible annotation delimiters | | other-control-format | any remaining Cc/Cf | catch-all for other control/format chars (TAB/LF/CR are treated as benign) |

API

const { scan } = require('unicode-inspector'); // or require('./scan.js')

const report = scan('a​b');
// report.findings -> [
//   { index: 1, codepoint: 0x200B, hex: 'U+200B',
//     name: 'ZERO WIDTH SPACE', class: 'zero-width' }
// ]
// report.counts   -> { 'zero-width': 1 }

• findings — one entry per suspicious codepoint, in order of appearance: { index, codepoint, hex, name, class }.
• counts — number of findings aggregated per class.

scan is pure: no DOM, no network, no filesystem. Non-string or empty input returns { findings: [], counts: {} }.

Index unit (pinned)

index is the codepoint index — the position of the character in [...str] (Array.from), not the UTF-16 code-unit offset. This is surrogate-pair safe: an astral character (for example an emoji) counts as a single position, so a hidden codepoint immediately after an emoji reports index === 1, not 2. Callers that need UTF-16 offsets must map codepoint indexes to code-unit offsets themselves.

Stripping / normalizing (strip.js)

strip() is the "clean it up" companion to scan(). It is built directly on top of scan.js (same pinned table in codepoints.js, same codepoint-aware indexing) — it never forks a second codepoint list, so what it acts on is exactly what scan() names.

const { strip } = require('unicode-inspector/strip.js'); // or require('./strip.js')

const { output, removed, kept } = strip('a​b'); // U+200B between a and b
// output  -> 'ab'
// removed -> [ { index: 1, codepoint: 0x200B, hex: 'U+200B',
//               name: 'ZERO WIDTH SPACE', class: 'zero-width', note: 'removed' } ]
// kept    -> []

• output — the cleaned string.
• removed — findings that were acted on (deleted, or for nbsp normalized), in order of appearance. Each has scan()'s shape ({ index, codepoint, hex, name, class }) plus a note describing the action.
• kept — findings whose class you toggled off: left byte-identical in output and reported here (same shape plus a note).

index on every finding is the codepoint index in the original input (the same unit scan() uses), so the audit stays aligned to your input regardless of how the output length shifts.

Three rules strip guarantees

1. Every invisible class is removed by default, and only those. The seven classes are zero-width, bidi-control, nbsp, soft-hyphen, variation-selector, interlinear-annotation, and other-control-format. Ordinary letters, digits, emoji, and the benign whitespace controls TAB/LF/CR (\t \n \r) are never touched — they pass through byte-identical, because scan() never flags them.

2. The NBSP family NORMALIZES; it does not delete. A no-break space and its relatives (U+00A0, U+2007, U+202F) are almost always meant to be a space — deleting one would silently join words. So by default nbsp is normalized to a single regular space (U+0020), not removed. A normalized NBSP still appears in removed (the string was transformed at that position), but its note says normalized to U+0020 (regular space) rather than removed. This is pinned as a code comment in strip.js too.

3. Idempotent. For a given set of options, strip(strip(x).output).output === strip(x).output: a second pass finds nothing left to act on (removed codepoints are gone; a normalized NBSP is now an ordinary U+0020, which the table does not flag).

Options — per-class toggles

Every class defaults ON (removed / normalized). Turn a class off to preserve those codepoints untouched and list them in kept instead:

// Keep intentional NBSP — e.g. French punctuation spacing puts a
// (narrow) no-break space before « ; : ! ? » and inside « … ».
strip('12 h', { keepNbsp: true });
//   -> output: '12 h' unchanged, kept: [ { ...class: 'nbsp',
//                                          note: 'kept (class disabled)' } ]

// Or disable any class by its exact name:
strip(src, { 'bidi-control': false }); // leave bidi controls in place
strip(src, { nbsp: false });           // same effect as keepNbsp: true

keepNbsp is a friendly alias for { nbsp: false }. It exists for the common, legitimate case where an NBSP is intentional (French typography, non-breaking "10 kg", etc.) and normalizing it would be wrong.

strip is pure: no DOM, no network, no filesystem. Non-string or empty input returns { output: '', removed: [], kept: [] } (a plain string input is returned unchanged as output).

Running the tests

npm test

This runs Node's built-in test runner (node --test) over test/*.test.js. It requires only Node.js (developed against Node 23; needs Node 18+ for the built-in test runner and Unicode property escapes). No dependencies to install.

The golden vectors in test/scan.test.js are hand-verified, with every hidden character written as a \u / \u{...} escape so the exact codepoint under test is reviewable by eye.

Confusable / homoglyph + mixed-script detection (confusables.js)

scan.js finds invisible codepoints; confusables.js finds deceptive visible ones — characters that look like other characters. This is the pаypal problem, where the а is Cyrillic (U+0430), not Latin a (U+0061): identical on screen, different bytes.

const { detectConfusables, skeleton } = require('unicode-inspector/confusables.js');

const r = detectConfusables('pаypal'); // the а is U+0430 (Cyrillic)
// r.findings -> [
//   { index: 1, codepoint: 0x0430, name: 'CYRILLIC SMALL LETTER A',
//     skeleton: 'a', looksLike: 'a' }
// ]
// r.mixedScripts -> [ { script: 'Latin', count: 5 }, { script: 'Cyrillic', count: 1 } ]

• findings — one entry per confusable character, in order of appearance: { index, codepoint, name, skeleton, looksLike }. index is the same codepoint index unit as scan() (surrogate-pair safe). name is the official Unicode name; looksLike is the character(s) it is confusable with; skeleton is that character's UTS #39 skeleton.
• mixedScripts — the whole-string mixed-script signal as [{ script, count }]. It is empty unless two or more writing systems co-occur among the string's letters. A non-empty array is the "this string mixes scripts" signal, sorted by count descending.

skeleton(str) exposes the UTS #39 skeleton directly: two strings are confusable when skeleton(a) === skeleton(b). For example skeleton('pаypal') === skeleton('paypal') === 'paypal'.

Baseline rule (why clean ASCII is always empty)

Plain ASCII (U+0000–U+007F) is treated as the reference alphabet — it is what other characters are confusable with — so ASCII characters are never themselves reported as findings. detectConfusables('microsoft') returns no findings even though the table maps m → rn. If you need ASCII↔ASCII look-alikes (rn vs m, l vs 1), compare skeleton() outputs of the two whole strings.

Two honest limits

1. Confusable detection is data-table-bound. A character is only flagged if it appears in the pinned Unicode confusables table (v17.0.0, UTS #39, 6565 mappings). Novel or un-tabled look-alikes are not caught, and the table reflects typical fonts — two glyphs "confusable" in the data may render distinctly in a given typeface.
2. Mixed-script is a heuristic. Genuinely multilingual text mixes scripts too (an English sentence quoting a Russian word). The strongest phishing signal is mixing within a single identifier-like token (a domain label, a username); this core reports the whole-string signal and leaves token-splitting policy to the caller. Only letters are counted; digits, punctuation, spaces, symbols and emoji are script-neutral and ignored.

The vendored Unicode data, its pinned version, and its license are documented in VENDOR.md.

Scope of these modules

Invisible-codepoint detection (scan.js), the strip/normalize transform (strip.js), and confusable/homoglyph + mixed-script detection (confusables.js). Deliberately out of scope here (built in later tasks): any UI.

License

MIT.