@mrok-creator/unifold

Zero-dependency, fully typed Unicode normalization for a data-intake / deduplication pipeline: sanitize strings for storage, build ephemeral matching keys, safely normalize URLs, and flag mixed-script ("homoglyph") domains.

• No runtime dependencies.
• Tree-shakeable (named exports only, sideEffects: false).
• Deterministic: no time, randomness, locale, or I/O in any normalization function.
• Every storage-level transform reports an audit trail (changes[]) of exactly what it changed.

Install

pnpm add @mrok-creator/unifold
# or
npm install @mrok-creator/unifold

Modules

sanitize — storage-level cleanup

Applies, in order: Cyrillic→Latin homoglyph folding, BOM removal, control-character→space replacement, zero-width-character removal, trim, and space-run collapsing. Returns the cleaned value plus an audit trail — this is the value you persist.

import { sanitize } from '@mrok-creator/unifold';

sanitize(' Offer  A ');
// { value: 'Offer A', changed: true, changes: [...] }

When you only need the cleaned string (no audit trail), use the value-only wrapper:

import { sanitizeValue } from '@mrok-creator/unifold';

sanitizeValue(' Offer  A '); // 'Offer A'

canonicalKey — matching-only key

Everything sanitize does, plus case folding and separator unification: general punctuation, dashes of all kinds, quotes of all kinds, underscore and NBSP all fold to a single space, so stylistic variants of one name converge to one key. Returns a plain string — never persist this, it's for comparison only.

import { canonicalKey } from '@mrok-creator/unifold';

canonicalKey('Offer-A') === canonicalKey('offer_a'); // true
canonicalKey('“Acme”–Co Ltd') === canonicalKey('Acme.Co Ltd'); // true → 'acme co ltd'

normalizeUrl — RFC 3986-safe URL normalization

Applies only transforms that are RFC 3986-safe or explicitly spec-mandated (duplicate-slash collapse in paths): trim + invisible-character removal, punctuation-junk stripping around a scheme:// URL, scheme/host lowercasing, default-port stripping, percent-encoding hex uppercasing (path only), and duplicate-slash collapsing (path only). Malformed input is returned as-is (with the safe text cleanups applied) — this function never throws.

import { normalizeUrl } from '@mrok-creator/unifold';

normalizeUrl('HTTP://Example.COM:80//a?utm=x');
// { value: 'http://example.com/a?utm=x', changed: true, changes: [...] }

normalizeUrl('".https://example.com".');
// { value: 'https://example.com', changed: true, changes: [{ rule: 'strip-junk', ... }] }

When you only need the normalized string (no audit trail), use the value-only wrapper:

import { normalizeUrlValue } from '@mrok-creator/unifold';

normalizeUrlValue('HTTP://Example.COM:80//a'); // 'http://example.com/a'

suspiciousDomain — mixed-script flag (detect only)

Flags a host whose letters mix scripts (e.g. Cyrillic look-alikes inside a Latin domain — pаypal.com vs paypal.com). Detection only: the host is never rewritten, because auto-fixing risks breaking a legitimate domain.

import { suspiciousDomain } from '@mrok-creator/unifold';

suspiciousDomain('pаypal.com');
// { host: 'pаypal.com', suspicious: true, reason: 'mixed-script', scripts: ['latin', 'cyrillic'] }

Note: pass the decoded Unicode host. Punycode (xn--) labels are not decoded, so an encoded look-alike domain will not be flagged. Opt-in punycode decoding is planned for a future release.

Rule tables

sanitize rule order (storage-level)

| Order | Rule id | What it does | | ----- | ----------------- | --------------------------------------------------------------------------------------------------------- | | 1 | homoglyph | Folds single-codepoint Cyrillic look-alikes to their Basic Latin letter (generated data, Unicode 16.0.0). | | 2 | bom | Removes a byte-order-mark (U+FEFF) anywhere in the string. | | 3 | control | Replaces control characters (U+0000–U+001F, U+007F) with a space. | | 4 | zero-width | Removes zero-width characters (e.g. ZWSP, ZWJ, ZWNJ). | | 5 | trim | Trims leading/trailing whitespace. | | 6 | collapse-spaces | Collapses runs of ASCII spaces (U+0020) to a single space; NBSP is never collapsed. |

canonicalKey runs all six of the above, then folds NBSP and the spec's separator punctuation to a space — general punctuation (. , ; : ! ? ( ) [ ] { } / \ | @ # $ % ^ & * + = ~), underscore, dashes of all kinds and quotes of all kinds — then case-folds, before a final trim + collapse. Stylistic variants of one name converge to a single key; characters outside the separator set pass through unchanged.

normalizeUrl rule order

| Order | Rule id | What it does | | ----- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | trim | Trims edge ASCII whitespace and NBSP (not native trim() — see "What it never touches"). | | 2 | invisible | Strips BOM, control characters and zero-width characters anywhere in the URL. | | 3 | strip-junk | Strips punctuation junk around a scheme:// URL: a leading non-alphanumeric run before the scheme, and trailing quotes / sentence punctuation (.,;:!) / unbalanced closing brackets — a balanced closer stays, so https://[::1] and …/Rust_(programming_language) are untouched while (https://a.com) sheds its wrapper. /, ? and # are never stripped from the end; plain non-URL strings, protocol-relative //host and alphanumeric prefixes (0https://…) are never touched. | | 4 | scheme-lowercase | Lowercases the scheme (HTTP:// → http://). | | 5 | host-lowercase | Lowercases the host. | | 6 | default-port | Strips an explicit default port (:80 for http, :443 for https). | | 7 | percent-encoding-uppercase | Uppercases percent-encoding hex triplets in the path (%2f → %2F). | | 8 | collapse-slashes | Collapses runs of / in the path to one; a run immediately after : is never collapsed, so :// survives even in junk-prefixed, mis-parsed input. |

What it never touches

• Query and fragment — never rewritten by the structural rules (no percent-encoding case changes, no slash collapsing). Edge text cleanup is the one exception: like trim with trailing whitespace, strip-junk removes trailing punctuation junk from the end of the string even when a query or fragment is present (…?q=hi. and …#frag. lose the final .; a trailing FQDN root dot in https://example.com. is stripped the same way).
• The www. prefix — not stripped or added; www.example.com and example.com normalize independently.
• Trailing slash — /a and /a/ are left exactly as given; no trailing-slash policy is imposed.
• Interior NBSP and underscores at the storage level — sanitize does not fold NBSP or dash/underscore variants; that folding is matching-only and happens in canonicalKey, never in the persisted value.

Type reuse

All public types are named exports re-exported from the package root, so consumers can type their own code against them instead of redeclaring shapes:

import type { NormalizationResult, SuspiciousDomainResult } from '@mrok-creator/unifold';

function persist(result: NormalizationResult): void {
  // result.value / result.changed / result.changes are fully typed
}

Full type surface: NormalizationResult, NormalizationChange, SanitizeRuleId, UrlRuleId, RuleId, SuspiciousDomainResult, SuspiciousReason.

Status

Published on npm. All four modules (sanitize, canonicalKey, normalizeUrl, suspiciousDomain) are implemented, tested (100% coverage on all four metrics), and reviewed. See CHANGELOG.md for releases and .claude/docs/ for the module map, architectural decision log, and Unicode data notes.