fix-encoding

A zero-dependency, MIT-licensed JavaScript library that reverses the single most common form of text corruption on the web: "mojibake" — the garbled é, ’, “ you see when accented characters and curly quotes get mangled. It is a small, honest ftfy-for-JS wedge: ftfy (the Python reference) fixes dozens of corruption families; this library nails the one that accounts for the overwhelming majority of real-world cases and refuses to touch anything else.

Pure function, string in / plain object out. No DOM, no network, no filesystem.

Intended package name: @verifyhash/fix-encoding. It is not published yet — this repo does not publish anything; graduation is an owner decision (see GO-LIVE.md).

The one bug it fixes

Almost all mojibake comes from a single mistake:

1. Text is correctly stored as UTF-8. The character é is the two bytes 0xC3 0xA9.
2. Some program reads those bytes as if they were Windows-1252 (cp1252) — the default legacy code page on Windows. 0xC3 renders as Ã, 0xA9 as ©, so é becomes é.
3. That wrong text is then re-saved as UTF-8, baking the corruption in.

fixEncoding reverses exactly that chain: it re-encodes the string back to cp1252 bytes and decodes those bytes as UTF-8.

| You have | You want | Cause | | --- | --- | --- | | café | café | é = C3 A9 read as cp1252 | | It’s | It's (curly ') | ' = E2 80 99 read as cp1252 | | “Helloâ€ | "Hello" (curly) | "/" read as cp1252 | | München | München | ü = C3 BC read as cp1252 | | 10–20 | 10–20 (en dash) | – read as cp1252 | | wait… | wait… (ellipsis) | … read as cp1252 |

It also recovers double-encoded text (the mistake applied twice, e.g. café) by repeating the repair until no mojibake evidence remains.

The CJK legacy chains (Shift_JIS / GBK / Big5 / EUC-KR)

The same UTF-8 → wrong-code-page → UTF-8 mistake also happens with the legacy East Asian code pages, and that garbling is what makes this library a genuinely differentiated ftfy-for-JS wedge rather than a curly-quote fixer.

Precisely: 日 is correctly stored as the three UTF-8 bytes E6 97 A5. A program reads those bytes as if they were Shift_JIS (or GBK / Big5 / EUC-KR). The double-byte legacy decoder re-groups the same bytes on different boundaries, so instead of one kanji you get a run of unrelated CJK glyphs and half-width katakana — the classic 譌･譛ｬ隱ｼ… garble seen in old CSV exports, mail archives and scraped pages. fixEncoding reverses it by re-encoding the garble back to the legacy code page's bytes and decoding those bytes as UTF-8.

| Language | Legacy code page | What UTF-8-mis-read-as-it looks like | | --- | --- | --- | | Japanese | Shift_JIS | 東京 → 譚ｱ莠ｬ | | Simplified Chinese | GBK | 中文 → 涓枃 | | Traditional Chinese | Big5 | 學校 → 摮郔恍 | | Korean | EUC-KR | 공원 → 怨듭썝-style glyphs |

(Node has no built-in TextEncoder for these code pages — it only encodes UTF-8 — so the library synthesises the legacy encoder by inverting the legacy decoder: it enumerates every byte / lead+trail pair the TextDecoder accepts and records char → bytes. That map is the exact right-inverse of the decoder, which is what makes the round trip provable.)

Environment-honesty law (feature detection)

Legacy decoders are only present when the JS runtime was built with the ICU data for them. This library never assumes a decoder exists: each label is feature-detected with new TextDecoder(label) inside a try/catch. Where a runtime lacks a given decoder, that chain is skipped and the skip is surfaced honestly in the returned steps (e.g. "1 legacy CJK decoder (euc-kr) is unavailable in this runtime, so that chain was skipped"). The library never throws because a decoder is missing.

Verified on this box: Node 18+ built with full ICU ships all four — shift_jis, gbk, big5, and euc-kr all construct and decode. The test suite asserts this (detectDecoders() returns all four available, none missing). A minimal-ICU or browser build may expose fewer; those chains are then reported as skipped rather than silently dropped.

Evidence gate for CJK (same discipline as the Latin chain)

A CJK candidate repair is accepted only when it strictly reduces a suspicion score — the count of codepoints that are rare in clean text but common in this garble: half-width katakana (U+FF61–FF9F), the Latin-1 supplement (U+0080–00FF), fullwidth ASCII, spacing-modifier letters, the Private Use Area, and the lossy U+FFFD (weighted heaviest). Concretely:

• The re-encoded bytes must be valid UTF-8 (a fatal decode must succeed) — the single strongest signal that the input really was UTF-8 mis-read as a legacy code page.
• Clean CJK text scores 0, so no repair can ever "reduce" it further: clean Japanese/Chinese/Korean is returned identity, changed:false.
• Ambiguous input that no chain strictly improves is left untouched at low confidence.
• The repair is idempotent — it stops at a suspicion fixed point, so fixEncoding(fixEncoding(x)) === fixEncoding(x).
• high confidence requires a byte-exact proof: re-applying the forward corruption to the recovered string reproduces your input exactly.

Honest limits: only text whose garble round-trips losslessly is recoverable. If the mis-decode hit an invalid legacy byte sequence (introducing U+FFFD), those bytes are gone and the chain declines rather than guessing. Chinese garble that happens to be all normal-looking ideographs (no half-width/symbol residue) carries no evidence, so it is conservatively left alone.

Who it's for

• Developers importing legacy CSV/JSON/SQL dumps where accents arrived garbled.
• Anyone building a paste-and-clean text tool, a scraper, or an ETL step.
• Log and email pipelines that keep surfacing ’ where an apostrophe belongs.

Install

npm install @verifyhash/fix-encoding

Zero dependencies. Ships CommonJS (require) with a bundled index.d.ts, so TypeScript callers get full types with no @types package. Node 18+.

Not yet published: this repo does not publish anything (see GO-LIVE.md). The scoped name above is the intended name and is reserved for the owner's npm publish --access public.

Usage

const { fixEncoding } = require('@verifyhash/fix-encoding');

fixEncoding('café');
// { output: 'café', changed: true, confidence: 'high', unrecoverable: 0,
//   steps: [ 'Inferred chain: the text was UTF-8, mis-read as Windows-1252
//             (cp1252), then re-saved as UTF-8.',
//            'Reversed the chain by re-encoding to cp1252 bytes and decoding
//             them back as UTF-8 (1 pass).' ] }

fixEncoding('The quick brown fox.');
// { output: 'The quick brown fox.', changed: false, confidence: 'low',
//   unrecoverable: 0, steps: [] }

API

fixEncoding(str: string) => { output: string, changed: boolean, confidence: 'high'|'medium'|'low', unrecoverable: number, steps: string[] }

• output — the repaired string (or the original, untouched).
• changed — true only if a repair was actually applied.
• confidence — how sure the repair is:
  • high — a repair was made, no mojibake residue remains, no lossy �, and the inferred chain round-trips byte-exact (re-applying the corruption the same number of passes reproduces your input exactly).
  • medium — evidence was reduced but the result is not fully certain: some residue remains, the round-trip is not byte-exact, and/or the text carries lossy � characters that make a byte-exact proof impossible.
  • low — nothing changed (clean input or no reversible chain) — ambiguous, so it was left alone.
• unrecoverable — count of � (U+FFFD) replacement characters: bytes that were already destroyed upstream and can never be recovered.
• steps — the repair chain list: a plain-English description of the inferred mis-decode chain, including a "twice" / "double-encoded" note when the corruption was applied more than once, plus an honest note when � losses are present (empty [] when nothing changed). Non-string input throws TypeError.

Refusal law. If there is no mojibake evidence, fixEncoding returns the input verbatim with changed: false and confidence: 'low' — it never touches clean text.

Idempotence. The function is a guaranteed fixed point on its own output: fixEncoding(fixEncoding(x).output).output === fixEncoding(x).output for every input x. Cleaned text carries zero evidence, so a second pass is always a no-op (asserted across every vector shape in the test suite).

Multi-pass and lossy text

Double- and triple-encoded text (the mistake applied more than once, e.g. café) is recovered by iterating the single-pass repair up to a hard ceiling of 3 passes, continuing only while mojibake evidence strictly decreases and stopping the instant it stops dropping. Inputs mangled beyond that ceiling are partially reduced and reported at medium confidence rather than guessed at.

When the input contains � (a byte already lost upstream), the text is split around each marker so the recoverable runs on either side are still repaired, the � markers are preserved verbatim in place, and the loss is reported in steps and unrecoverable. No bytes are ever invented to fill the gap.

Honesty laws

This library is deliberately narrow and conservative:

• Heuristic, not a proof. It repairs the dominant UTF-8→cp1252→UTF-8 chain plus the four legacy CJK chains (UTF-8 mis-read as Shift_JIS / GBK / Big5 / EUC-KR — see the section above). Other exotic corruption (KOI8-R, partial fixes) is out of scope and is left untouched rather than guessed at.
• It refuses clean text. Before touching anything it looks for mojibake evidence — a lead artifact (Ã / Â / â) immediately followed by a character that is the cp1252 rendering of a UTF-8 continuation byte. A repair is accepted only if it strictly reduces that evidence count; otherwise the input is returned verbatim with changed:false. Correctly-encoded accented prose (café, “quoted”, €9) has no such artifact and is never altered.
• It is idempotent. Running it on already-clean output is a guaranteed no-op — cleaned text has zero evidence, so the gate declines.
• U+FFFD is lossy — and honored as such. The Unicode replacement character � means a byte was already destroyed upstream; that information cannot be recovered. This library never introduces new � to force a "fix" (a candidate repair that adds replacement characters is rejected), and it never fabricates the lost bytes. Recoverable text on either side of a � is still repaired, the marker is preserved in place, and the loss is reported in steps / unrecoverable; such inputs can never earn high confidence.

Known limits

• Text that mixes clean and corrupted runs is repaired per-run: only the byte spans that form a valid UTF-8 mojibake sequence are reversed, and clean cp1252-special characters embedded in the same string — a lone em dash, curly quote, euro sign or ellipsis — are preserved verbatim rather than clobbered. (Earlier a single clean em dash beside a garbled word could block the whole repair; that is fixed and pinned by the adversarial suite.) Because such an embedded clean character breaks the byte-exact round-trip proof, a correct repair of that kind is reported at medium confidence, not high. Very unusual clean strings that happen to contain a literal Ã+continuation pair (rare in natural language) could still be misread — measure on your own corpus.
• The repair never introduces a C1 control character (U+0080..U+009F). A stray lead byte followed by a continuation byte that would decode to a non-printable control code is lead-byte bait, not real text, so that pass is rejected and the input is left unchanged (the same discipline as the U+FFFD guard). Genuine multi-pass reversals only ever carry a C1 code forward from an undefined cp1252 slot and consume it in a later pass, so this never blocks a real double/triple-encode recovery.
• The single-byte source code pages modeled are cp1252/Latin-1; the double-byte ones are Shift_JIS, GBK, Big5 and EUC-KR. Corruption originating from other code pages (KOI8-R, ISO-8859-x variants beyond Latin-1, EUC-JP, …) is not addressed.
• CJK garble is only recovered when it round-trips losslessly. A mis-decode that introduced U+FFFD upstream is unrecoverable, and garble consisting solely of normal-looking ideographs carries no evidence and is left alone.

Running the tests

cd fix-encoding
npm test

Uses Node's built-in test runner (node --test, Node 18+); no dependencies to install. The golden vectors are generated in-test: each one starts from a known-clean string, is encoded to real UTF-8 bytes, and those bytes are decoded as cp1252 to synthesize the garble mechanically — so the suite can never drift onto a mis-remembered mojibake spelling. Tests cover the canonical repair, French/German/Spanish accents, curly quotes, dashes, the ellipsis, the euro sign, double- and triple-encoding recovered within the 3-pass ceiling, the depth cap on over-encoded input, the high/medium/low confidence ratings, U+FFFD loss handling (recoverable text preserved, loss reported), idempotence on every vector shape, and the clean/empty identity cases. A dedicated test/adversarial.test.js hardens the core against hostile-but-real inputs: dense-diacritic false positives (Vietnamese/Portuguese, currency and box-drawing symbols), garble embedded beside clean cp1252-special characters, mixed double-encode + U+FFFD chains in one string, 100k+ char scale, pure lead-byte bait, and the steps/confidence contract.

License

MIT — see LICENSE.