distillate

Probabilistic data structures for JavaScript: space-efficient, approximate answers with tunable error and zero false negatives. TypeScript-first, zero dependencies, and the right structure per workload. It opens with a family of membership filters and is built to grow into other sketches.

Pre-release (0.x). The published structures are correct, tested, and benchmarked, but the public API may still change before 1.0. Pin a version if you depend on it.

Why

An approximate-membership query (AMQ) filter answers "is this in the set?" with a tunable false-positive rate and zero false negatives, in a fraction of the space of storing the set itself.

• Runs anywhere: Node, Bun, Deno, browsers, Cloudflare/Vercel edge.
• Correct: no false negatives, property-tested.
• Small: per-structure subpath imports, zero runtime dependencies.
• Portable: a versioned binary format (toBytes / fromBytes).

Why these promises, and what the incumbents get wrong: overview.md.

Install

npm install distillate
# or: pnpm add distillate / bun add distillate / deno add npm:distillate

Requires Node 22+ (or any modern Bun/Deno/browser/edge runtime).

Runtime support

distillate targets ES2022 with zero runtime dependencies and no eval, so it runs on every modern JavaScript runtime:

• Node.js 22, 24 (LTS and current)
• Bun and Deno
• Browsers and Cloudflare/Vercel edge

Every push runs a CI smoke matrix that imports the built package on Node 22/24, Bun, and Deno, so cross-runtime support is verified, not assumed.

Structures

Each structure ships as its own subpath, so you only bundle what you import.

| Import | Structure | Answers | Use for | | --------------------- | -------------- | ------------------- | -------------------------------------------------------- | | distillate/bloom | Classic Bloom | seen this key? | Familiar default, migration from bloom-filters | | distillate/scalable | Scalable Bloom | seen this key? | Key count unknown or growing; keeps its FPR bound | | distillate/blocked | Blocked Bloom | seen this key? | Faster lookups and a lower FPR for a small space premium | | distillate/fuse | Binary Fuse | seen this key? | Static set built once and queried a lot; least space | | distillate/cuckoo | Cuckoo | seen this key? | Keys come and go; the one filter with delete | | distillate/hll | HyperLogLog | how many distinct? | Counting distinct users, IPs, or keys in fixed space | | distillate/countmin | Count-Min | how many times? | Counting events per key in fixed space | | distillate/topk | Top-K | which are heaviest? | Finding the most frequent keys in fixed space |

The filters are mutable except Binary Fuse, which is built once from the whole key set. HyperLogLog, Count-Min and Top-K are sketches rather than filters: none reports whether it saw any particular key. HyperLogLog counts distinct keys, Count-Min counts how often each key appeared, and Top-K reports which keys appeared most.

Classic Bloom (distillate/bloom)

import { BloomFilter } from "distillate/bloom";

const filter = BloomFilter.create(100_000, 0.01); // capacity, target FPR
filter.add("alice");
filter.has("alice"); // true
filter.has("bob"); // false (or a ~1% false positive)

const bytes = filter.toBytes();
const restored = BloomFilter.fromBytes(bytes);

Also: union(other) (merge equal-parameter filters), bitsPerKey, and a low-level new BloomFilter({ m, k, seed }).

Scalable Bloom (distillate/scalable)

For when you cannot say how many keys are coming. It grows by opening larger, tighter Bloom stages as keys arrive, and keeps its false-positive rate under the target however many open. Full API and trade-offs: Scalable Bloom guide.

import { ScalableBloomFilter } from "distillate/scalable";

const seen = ScalableBloomFilter.create(10, 0.01); // first stage, target FPR
for (let i = 0; i < 30; i++) seen.add(`key-${String(i)}`);

seen.stages; // 2
seen.has("key-29"); // true

A key already held is not counted again, so duplicates never use up capacity. It costs more bits per key than a Classic Bloom sized for a known n; if you know n, use that.

Also: from(keys, epsilon), union, equals, rate, toBytes / fromBytes, toJSON / fromJSON, and growth / tightening / seed options.

Blocked Bloom (distillate/blocked)

import { BlockedBloomFilter } from "distillate/blocked";

const filter = BlockedBloomFilter.create(100_000, 0.01);
filter.add("alice");
filter.has("alice"); // true

Same surface as Classic Bloom (add / has / union / toBytes / fromBytes / bitsPerKey). Reach for it when lookup throughput matters; prefer Classic when space is tight. Cache-line rationale, speed ratios, and the space penalty: Blocked Bloom guide.

Binary Fuse (distillate/fuse)

A static filter: built once from the full key set, then immutable. The most space-efficient option in the lineup. Bits-per-key, FPR, and query throughput: Binary Fuse guide.

import { BinaryFuse8, BinaryFuse16 } from "distillate/fuse";

const filter = BinaryFuse8.from(["alice", "bob", "carol"]);
filter.has("alice"); // true
filter.size; // 3
filter.bitsPerKey; // 64

// Lower false-positive rate, twice the space:
const precise = BinaryFuse16.from(["alice", "bob", "carol"]);

bitsPerKey is the space actually allocated, not the asymptotic figure. Three keys pay 64 bits each because the fingerprint array has a fixed minimum; the ~9 you see quoted is what a filter approaches once n is large (about 9.5 at 100k). Size honestly with fuseBitsPerKey(n, width), or the sizing guide.

Also: toBytes / fromBytes. No add / delete; rebuild from the new set to change membership.

Cuckoo (distillate/cuckoo)

For a set that shrinks as well as grows: the one filter here with delete. Full API and trade-offs: Cuckoo guide.

import { CuckooFilter } from "distillate/cuckoo";

const sessions = CuckooFilter.create(100_000, 0.01);
sessions.add("alice");
sessions.has("alice"); // true

sessions.delete("alice"); // true
sessions.has("alice"); // false

Only delete keys you added: a key never added can share a fingerprint with one that was, and deleting it removes that one instead. An add into a full filter throws CuckooFullError and leaves the filter unchanged. It is larger than a Classic Bloom at common targets and smaller only at about 0.2% and below, so pick it for delete, not for space.

Also: from(keys, epsilon), equals, rate, count, toBytes / fromBytes, toJSON / fromJSON, cuckooSizing(n, epsilon), and a seed option.

HyperLogLog (distillate/hll)

A sketch, not a filter: it counts distinct keys in space fixed by precision rather than by the answer, and cannot report whether it saw any particular key. 12 KiB counts a thousand distinct keys or a billion, at about 0.8% relative error. Full API and sizing: HyperLogLog guide.

import { HyperLogLog } from "distillate/hll";

const sketch = HyperLogLog.create(0.01); // target relative error
sketch.add("alice");
sketch.add("bob");
sketch.add("alice"); // already counted

sketch.count(); // 2

// Merge per-shard sketches without double-counting the overlap:
const other = HyperLogLog.from(["bob", "carol"], 0.01);
sketch.union(other).count(); // 3

Below a few thousand distinct keys the sketch counts rather than estimates, so small answers are exact. It switches to fixed-size registers on its own once that stops paying, with nothing to configure.

Also: equals, toBytes / fromBytes, toJSON / fromJSON, standardError, hllSizing(relativeError), and a low-level new HyperLogLog({ p, seed }).

Count-Min (distillate/countmin)

A sketch, not a filter: it estimates how many times each key appeared, in space fixed by the error you ask for rather than by how many keys arrive. The estimate is never below the truth, and at most epsilon of the total above it. Full API and sizing: Count-Min guide.

import { CountMinSketch } from "distillate/countmin";

const hits = CountMinSketch.create(0.001, 0.001); // error factor, failure probability
hits.add("/login");
hits.add("/login", 4);
hits.add("/signup");

hits.count("/login"); // 5
hits.total; // 6

// Combine per-shard sketches: counts add, exactly as if one sketch saw both streams.
const other = CountMinSketch.from(["/login", "/login"], 0.001, 0.001);
hits.union(other).count("/login"); // 7

from counts repeats, unlike CuckooFilter.from which ignores them: for a frequency sketch the repeats are the measurement. A counter that would pass 2^32 - 1 throws CountMinOverflowError rather than wrapping, since a wrapped counter would read as an underestimate.

Also: equals, toBytes / fromBytes, toJSON / fromJSON, error(), epsilon, delta, width, depth, countMinSizing(epsilon, delta), and a low-level new CountMinSketch({ width, depth, seed }).

Top-K (distillate/topk)

A sketch that lists the keys seen most often, in a table fixed by the error you ask for rather than by how many keys arrive. A key it holds never reads below its true count, and any key heavier than error() is guaranteed to be held. Full API and the bound: Top-K guide.

import { TopK } from "distillate/topk";

const routes = TopK.create(0.01); // error factor
routes.add("/login", 5);
routes.add("/signup", 2);
routes.add("/about");

const decoder = new TextDecoder();
routes
  .top(2)
  .map((e) => decoder.decode(e.key))
  .join(); // "/login,/signup"
routes.count("/login"); // 5

Keys come back as Uint8Array, exactly as recorded, so a serialized Top-K sketch holds your keys verbatim, unlike every other frame here. union combines sketches soundly but, unlike Count-Min's, not byte for byte as one sketch fed both streams.

Also: equals, toBytes / fromBytes, toJSON / fromJSON, total, error(), epsilon, capacity, topKSizing(epsilon), and a low-level new TopK({ capacity, seed }).

Performance

Classic Bloom head-to-head at a matched 1% false-positive rate over the same 100k keys, measured by identical code (cross-library harness, node v24.14.1, Apple M5):

| Classic Bloom | bits/key | measured FPR | has throughput | | -------------- | -------- | ------------ | ---------------- | | distillate | 9.59 | 1.01% | ~21.8 M ops/s | | bloom-filters | 9.59 | 0.99% | ~0.29 M ops/s |

Same space, same accuracy, ~75x the lookup throughput of bloom-filters (the package distillate replaces), while hashing UTF-8 bytes with MurmurHash3 so filters stay portable and cross-language readable.

Cardinality is a separate head-to-head, at a matched register count (m = 2 ** p, p = 14, the configuration Redis uses) over the same keys, swept from 1k to 10M distinct:

| p = 14, m = 16384 | distillate | bloom-filters | | -------------------- | --------------- | ---------------- | | rel. error, n=1k | 0.00% | 49.63% | | rel. error, n=100k | 0.82% | 0.78% | | rel. error, n=10M | 0.15% | 0.41% | | serialized, n=10M | 12,314 B binary | 40,247 B JSON | | build 10M keys | 0.63 s | 1,165 s (19 min) | | sustained add, n=10M | ~16 M ops/s | ~9 k ops/s |

Both sketches carry the same theoretical error at this precision (1.04 / sqrt(m), about 0.81%), and once n is well past m both stay inside it. The differences are elsewhere.

Small counts. bloom-filters has no working small-range correction, so below roughly 2.5 * m it is off by about half: counting a few thousand distinct keys in a 16k-register sketch, it answers 504 for 1,000. distillate is exact there because it is still holding sparse entries, not because its estimator is better.

Memory does not grow. From 10k distinct keys to 10M, distillate stays at exactly 12,314 bytes. The incumbent's JSON grows from 32,904 to 40,247 bytes over the same range, because larger register values take more digits to spell out.

Building the sketch. bloom-filters holds a flat 9k adds/sec at every size, so its cost is purely linear: 10.6 seconds for 100k distinct keys, 109 seconds for 1M, and 1,165 seconds for 10M. distillate counts the same 10M in 0.63 seconds, and its rate climbs with n rather than staying flat, reaching about 16 M ops/s once the registers are dense.

These are a point-in-time snapshot on one machine. The full report (blocked/fuse, 1M capacity, the bloomfilter micro-package) and exactly how it is measured live in the apps/bench workspace: RESULTS.md, METHODOLOGY.md.

Docs

Design notes, the structure decision matrix, hashing, and the binary format live in docs/:

• overview: what and why
• choosing a structure: decision matrix and the full lineup
• serialization: the binary format spec
• versioning: SemVer policy and supported-runtime baseline
• architecture, hashing: contributor notes, on GitHub
• API reference: generated from TSDoc at site build time (per entry point)

License

MIT © Akshay