zstd-js

Zstandard compression and decompression in pure JavaScript. No WebAssembly, no native bindings — it runs anywhere JavaScript does, including React Native and Hermes.

A complete Zstandard codec with no dependencies and no reliance on Node's Buffer. Compression does LZ77 matching, Huffman-coded literals, FSE-coded sequences with custom tables, and repeat offsets. Decompression reads frames from any encoder, including features this one never emits. Both directions stream, and both support dictionaries. Every frame is verified against libzstd.

Install

npm install zstd-js

Usage

const zstd = require('zstd-js');

const frame = zstd.compress('hello world');
const back = zstd.decompress(frame);   // <Buffer 68 65 6c 6c 6f ...>

compress accepts a string, Buffer, TypedArray, DataView or ArrayBuffer, and returns a standard .zst frame. Any Zstandard decoder reads it — the zstd CLI, Node's built-in zlib.zstdDecompressSync, fzstd, or this package's own decompress.

Results are a Node Buffer where the runtime has one, and a plain Uint8Array otherwise. A Buffer is a Uint8Array, so the same code works in both; nothing internally depends on Buffer existing, so there is no polyfill to install in a browser or under Hermes.

zstd.compress(data, {
  checksum: true,        // append the XXH64 content checksum (4 bytes)
  windowSize: 1 << 20,   // farthest a match may reach back, default 4 MiB
  dictionary: dict,      // see below

  // Effort knobs. See "On compression levels" below before reaching for them.
  searchDepth: 32,       // candidates examined per position
  maxMisses: 16,         // consecutive first-byte failures before giving up
  goodEnough: 64         // match length at which the search stops early
});

zstd.decompress(frame, { dictionary: dict });

Dictionaries

Dictionaries help a great deal on small payloads that share structure, because matches may reach into them. Both kinds are supported:

// Any buffer of representative data works as a raw-content dictionary
const dict = require('fs').readFileSync('samples.bin');

const frame = zstd.compress(payload, { dictionary: dict });
const back = zstd.decompress(frame, { dictionary: dict });

A dictionary trained by the reference tool works too, and is usually better:

zstd --train samples/* -o trained.dict

A trained dictionary carries its own entropy tables, starting repeat offsets and an identifier; the identifier is written into the frame so a decoder can tell which dictionary it needs. On a 1.8 KB JSON record, a trained dictionary took the output from 237 bytes to 66.

Either way, the frame can only be read by a decoder holding the same dictionary — this package, libzstd, or zstd -d -D trained.dict. Decoding without it fails rather than returning wrong bytes.

Streaming

Neither the whole input nor the whole output has to be in memory:

const { Compress, Decompress } = require('zstd-js');

const parts = [];
const stream = new Compress((chunk, final) => parts.push(chunk));
stream.push(firstChunk);
stream.push(secondChunk);
stream.end();

const out = [];
const decoder = new Decompress((chunk, final) => out.push(chunk));
decoder.push(frameBytes, true);

Both take the same options as their one-shot counterparts. Compress also accepts streamHistory, the number of already-emitted bytes kept available for later blocks to match against; it defaults to one block, 128 KB, and setting it to 0 matches each block on its own.

Several frames at once

A Zstandard stream may hold frames back to back, and may carry skippable frames of user metadata. decompress walks the whole stream, joining the content and stepping over anything skippable.

TypeScript definitions ship with the package.

On compression levels

There is deliberately no level option. Levels imply that asking for more effort yields a smaller result, and that is not true of this parser. Measured across a mixed corpus, raising every effort knob together — search depth 4 to 256 — moved total output by well under a percent in both directions while costing six times the time:

| effort | speed | total output | |---|---|---| | lowest | 30 MB/s | 307,769 | | low | 30 MB/s | 298,066 | | default | 22 MB/s | 303,013 | | high | 12 MB/s | 303,189 | | highest | 5 MB/s | 304,010 |

The match finder already finds the longest matches available; the gap to real zstd is in which matches get chosen, not how hard it looks for them. The knobs are exposed for anyone who wants to trade time for a specific corpus, but they are not a level scale and are not documented as one.

Why

Every Zstandard implementation for JavaScript is either a native binding or a WebAssembly build. That leaves two gaps:

• React Native. Hermes has no WebAssembly, so none of the WASM packages run there.
• Synchronous APIs. WASM modules need asynchronous initialisation, which cannot back a compressSync.

Decoding was already solved in pure JS by fzstd. Encoding was not — there is no other pure-JavaScript Zstandard compressor on npm. That is what this package adds.

Decoding started out delegated to fzstd, but dictionary support needed a decoder that could be seeded with dictionary content, so it is now implemented here. The package has no dependencies.

Benchmarks

Compression

Measured against Node's native Zstandard (libzstd) and gzip:

| Input | Original | zstd-js | zstd | gzip | vs zstd | |---|---|---|---|---|---| | English text | 900,000 | 139 | 140 | 2,698 | 0.99x | | HTML | 840,000 | 133 | 133 | 2,520 | 1.00x | | Prose | 545,025 | 165,292 | 145,428 | 135,544 | 1.14x | | CSV | 184,579 | 33,986 | 30,260 | 41,935 | 1.12x | | JSON | 907,781 | 50,173 | 27,310 | 102,228 | 1.84x | | Incompressible | 900,000 | 900,031 | 900,030 | 900,293 | 1.00x |

Compression runs at roughly 11-30 MB/s on dense data and 180 MB/s on repetitive data, and around 125 MB/s on data it recognises as incompressible, which it detects and passes through rather than searching.

The corpus is generated by scripts/benchmark.js rather than read from disk, so these numbers are reproducible, and a test fails if this package's column drifts from what the code actually produces. The zstd and gzip columns come from the libraries Node was built against and vary a little between releases; these were measured on Node 22.

Decompression

Against fzstd, the other pure-JS Zstandard decoder, decoding frames produced by libzstd:

| Input | zstd-js | fzstd | ratio | |---|---|---|---| | English text | 5,921 MB/s | 917 MB/s | 6.45x | | HTML | 7,598 MB/s | 912 MB/s | 8.33x | | Prose | 234 MB/s | 199 MB/s | 1.17x | | CSV | 215 MB/s | 204 MB/s | 1.05x | | JSON | 547 MB/s | 347 MB/s | 1.58x | | Incompressible | 5,925 MB/s | 4,220 MB/s | 1.40x |

Faster on all six, and several times faster where matches dominate.

Unlike the compression table above, these figures are a point-in-time measurement: fzstd is not a dependency, so the build cannot recheck them.

Roadmap

• [x] Frame header: single-segment and explicit Window_Descriptor paths
• [x] Block framing: Raw_Block and RLE_Block
• [x] Bitstream writer and reader, with zstd's backward-read convention
• [x] Verified code tables and predefined FSE distributions
• [x] LZ77 match finder, hash chains with configurable search depth
• [x] FSE encoder, predefined tables
• [x] Compressed_Block assembly, with fallback to raw when it would not help
• [x] Huffman literal coding, with the four-stream layout
• [x] FSE encoder, custom tables with normalisation and table transmission
• [x] Repeat offsets
• [x] Lazy matching
• [x] xxhash64 content checksum, one-shot and incremental
• [x] Streaming API for both directions
• [x] Dictionary support, both directions
• [x] Cross-block matching, one-shot and streaming
• [x] Decoder, replacing the last dependency
• [x] Word-at-a-time bit extraction and an inlined Huffman loop
• [x] Incompressible input detected and passed through
• [x] Runs without Node's Buffer, so browsers and Hermes need no polyfill
• [x] exports map, so bundlers and TypeScript resolve it directly
• [x] Multi-frame and skippable-frame decoding
• [x] Trained dictionary format, both directions
• [x] Table modes priced against each other rather than picked by rule
• [x] Profiled and optimised: compression more than twice as fast, decompression around 60% faster
• [ ] Optimal parsing, to close the gap on JSON

Design notes

Everything is written against RFC 8878, and the code tables carry self-checks: the predefined distributions must sum to 2^accuracy_log, and the literal-length and match-length baselines must be contiguous under their extra-bit widths. Those checks caught a transcription error during development.

The bitstream is the part most likely to be subtly wrong, so it is fuzzed: 5,000 randomised field sequences are written and read back per test run.

Releasing

Releases are automated. Either run the Release workflow from the Actions tab and pick a bump, or tag locally:

npm version patch   # bumps and tags, with no v prefix
git push --follow-tags

Pushing the tag runs the suite, publishes to npm with a provenance attestation, and opens the GitHub release. Publishing is authenticated by OpenID Connect through a trusted publisher on npmjs.com, so no token is stored in this repository.

Testing

npm test

Every frame produced is round-tripped through Node's native Zstandard, which is libzstd itself, so correctness is measured against the reference implementation rather than against this package's own decoder. The suite covers every input length from 0 to 200, the 128 KB block boundaries, text, JSON, CSV, source, incompressible and mixed content, randomised payloads over restricted alphabets, and frames produced by libzstd at every compression level from 1 to 22.

Limitations

• JSON-like input compresses about 1.9x larger than real zstd. The match finder already finds the longest matches available, so the gap is in which matches are chosen rather than which exist. Closing it needs a parser that prices whole paths instead of deciding position by position; a first attempt at one came out worse than the current greedy-plus-lazy parser and is not shipped.
• There is no level option, for the reason above.
• Trained dictionaries are used for their content, repeat offsets and identifier. Their entropy tables seed the repeat modes but the encoder does not yet emit those modes itself, so a little of their benefit is unused.

License

MIT