@nxtedition/cache

A two-tier async cache with SQLite persistence, in-memory pseudo-LRU, stale-while-revalidate, cross-thread deduplication via SharedArrayBuffer, and automatic request coalescing.

Features

• Two-tier storage — In-memory cache backed by SQLite on disk
• File-sharded SQLite — Keys are hash-routed across N independent SQLite files, bypassing SQLite's per-file writer serialization. ~50% higher throughput under multi-thread write contention (see Benchmarks).
• Stale-while-revalidate — Serve stale data synchronously while refreshing in the background
• Request coalescing — Concurrent fetches for the same key share a single in-flight Promise
• Cross-thread locking — SharedArrayBuffer + Atomics.compareExchange / Atomics.waitAsync prevent redundant valueSelector calls across worker threads in the same process that share the same location
• Async value resolution — Transparently fetches missing values via a user-defined valueSelector
• Binary support — Store and retrieve Buffer / Uint8Array alongside JSON values
• Size-bounded storage — Configurable max database size with automatic eviction of oldest entries
• Custom serialization — Pluggable serialize/deserialize for non-JSON value types
• Batched writes — SQLite writes are coalesced into transactions via setImmediate, reducing I/O
• Disposable — Implements Symbol.dispose for use with using declarations

Usage

import { Cache } from '@nxtedition/cache'

const cache = new Cache(
  './my-cache.db', // SQLite file path, or ':memory:'
  async (id: string) => {
    const res = await fetch(`https://api.example.com/items/${id}`)
    return res.json()
  },
  (id: string) => id, // keySelector: derive cache key from arguments
  {
    ttl: 60_000, // 60 s before value is considered stale
    stale: 30_000, // serve stale for 30 s while revalidating
  },
)

const result = cache.get('item-123')

if (result.async) {
  // Cache miss — value is being fetched
  const value = await result.value
} else {
  // Cache hit — value returned synchronously
  const value = result.value
}

Using Symbol.dispose

{
  using cache = new Cache('./tmp.db', fetchItem, (id) => id, { ttl: 5_000 })
  const result = cache.get('key')
  // ...
} // cache.close() called automatically

API

new Cache(location, valueSelector?, keySelector?, opts?)

| Parameter | Type | Description | | --------------- | ---------------------------------- | --------------------------------------------- | | location | string | SQLite database path, or ':memory:' | | valueSelector | (...args) => V \| PromiseLike<V> | Function to fetch a value on cache miss | | keySelector | (...args) => string | Function to derive a cache key from arguments | | opts | CacheOptions<V> | Optional configuration |

CacheOptions

| Option | Type | Default | Description | | ------------ | ---------------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | ttl | number \| (value, key) => number | MAX_SAFE_INTEGER | Time-to-live in milliseconds. After this, the entry is stale. | | stale | number \| (value, key) => number | MAX_SAFE_INTEGER | Stale-while-revalidate window in ms. After ttl + stale, the entry is purged. | | memory | MemoryOptions \| false \| null | { maxSize: 16MB, maxCount: 16384 } | In-memory cache config, or false/null to disable. | | database | DatabaseOptions \| false \| null | { timeout: 20, maxSize: 128MB } | SQLite config, or false/null to disable persistence. | | lock | false \| null | enabled | Pass false/null to disable cross-thread SAB locking (see Cross-Thread Locking). Always off for ':memory:'. | | serializer | Serializer<V> | JSON + ArrayBufferView passthrough | Custom { serialize, deserialize } for value encoding. |

MemoryOptions

| Option | Type | Default | Description | | ---------- | -------- | -------------------------- | ---------------------------------------------- | | maxSize | number | 16 * 1024 * 1024 (16 MB) | Maximum total size in bytes of cached entries. | | maxCount | number | 16 * 1024 (16384) | Maximum number of entries in memory. |

The in-memory tier uses a random-two-choice eviction strategy: when the cache is full, two entries are sampled at random and the least recently accessed one is evicted. This provides near-LRU behavior with O(1) eviction cost.

DatabaseOptions

| Option | Type | Default | Description | | --------- | -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | timeout | number | 20 | SQLite busy timeout in milliseconds. | | maxSize | number | 128 * 1024 * 1024 (128 MB) | Maximum total database size across all shards. Divided evenly per shard. Oldest entries are evicted when a shard is full. | | shards | number | 4 | Number of SQLite files (shards) the cache is spread across. See File Sharding. Use 1 for single-file. |

Serializer<V>

| Method | Signature | Description | | ------------- | ---------------------------------------------- | --------------------------- | | serialize | (value: V) => Buffer \| Uint8Array \| string | Encode a value for storage. | | deserialize | (data: Buffer \| string) => V | Decode a stored value. |

The default serializer passes ArrayBufferView values through as-is and uses JSON.stringify/JSON.parse for everything else.

CacheResult<V>

Both get() and peek() return a CacheResult<V>, a discriminated union on the async property:

| async | value | Meaning | | ------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | false | V \| undefined | Cache hit — the value is available synchronously. Also returned for stale entries (a background refresh is triggered). undefined when peek() has no cached entry. | | true | Promise<V> | Cache miss — value is a Promise that resolves when the valueSelector completes. |

const result = cache.get('key')

if (result.async) {
  const value = await result.value // miss — await the fetch
} else {
  const value = result.value // hit (fresh or stale) — use directly
}

Methods

cache.get(...args): CacheResult<V>

Returns a cached value or triggers a fetch on cache miss. If the entry is stale and the valueSelector is async, returns the stale value synchronously (async: false) while a background refresh runs. If the valueSelector throws during a stale revalidation, the error is emitted and the stale value is preserved.

cache.peek(...args): CacheResult<V>

Same as get() but does not trigger a refresh on cache miss or stale entry. Returns { value: undefined, async: false } for missing/expired entries, or the stale value if within the stale window.

cache.refresh(...args): CacheResult<V>

Forces a new fetch via valueSelector regardless of cache state. Unlike get(), concurrent refresh() calls for the same key do not deduplicate — each call invokes the valueSelector. However, get() calls during a pending refresh() will return the in-flight promise.

cache.delete(...args): void

Remove an entry from both memory and SQLite. Also cancels any in-flight deduplication for that key — a pending fetch will still resolve for its callers, but the result is not written to the cache.

cache.gc(): void

Remove all expired entries (past ttl + stale) from both the in-memory cache and SQLite, and run PRAGMA wal_checkpoint(TRUNCATE) + PRAGMA optimize.

cache.flushSync(): void

Synchronously drain all pending batched writes to SQLite. The cache remains open and usable afterwards. This is useful when you need to guarantee persistence at a specific point without closing the cache (e.g. before handing off to another cache instance that shares the same database).

cache.close(): void

Calls flushSync() to drain pending writes, then closes the SQLite database and releases resources. Clears all in-flight deduplication. Operations after close() throw.

Also available as [Symbol.dispose]() for use with using declarations.

Open caches are automatically closed on the beforeExit event, ensuring pending writes are flushed before the process exits.

cache.stats

Returns runtime statistics:

{
  dedupe: { size },
  memory: { size, maxSize, count, maxCount } | undefined,
  database: { location, size } | undefined,
}

Deduplication

Concurrent calls to get() for the same key share a single in-flight Promise. The valueSelector is called only once:

// valueSelector is called once, both promises resolve to the same value
const [a, b] = await Promise.all([cache.get('key').value, cache.get('key').value])

If a fetch fails, the deduplication entry is cleaned up and subsequent calls retry.

Calling cache.delete(key) while a fetch is in-flight invalidates the deduplication entry. The pending promise still resolves for its callers, but the result is not written to the cache.

refresh() does not deduplicate with itself — each call starts a new fetch. However, get() calls see the most recent pending promise.

Stale-While-Revalidate

When an entry's TTL has expired but is still within the stale window, get() returns the stale value synchronously (async: false) and triggers a background refresh (when the valueSelector is async). If the refresh fails, the stale value is preserved.

Once the stale window expires, the entry is purged entirely and the next get() returns async: true.

|--- ttl ---|--- stale ---|
    fresh        stale      expired
      ↓            ↓           ↓
  sync hit     sync hit    async miss
             + bg refresh

Cross-Thread Locking

Worker threads in the same process that pass the same location to new Cache(...) share a 256 KiB SharedArrayBuffer (64K Int32 slots, acquired via @nxtedition/shared's getOrCreate registry). That buffer is treated as a hash table of per-key binary mutex slots:

• Acquire: Atomics.compareExchange(slot, 0, 1). The winner runs valueSelector; losers call Atomics.waitAsync (with a 1 s timeout as a guard against missed notifies and holder crashes). On timeout, the waiter falls back to a lockless refresh.
• Release: The holder does Atomics.sub(slot, 1) + Atomics.notify inside the batched flush, waking waiters which then read the freshly-cached value from the database.
• Exception safety: All exception paths (sync valueSelector throw, async rejection, serializer/ttl/stale throwing inside #set) release the slot under a try/finally, so a buggy user callback can never wedge a slot.

When the location is ':memory:', the cache is inherently per-instance (the SQLite DB isn't shared), so the SAB is skipped entirely and all coordination falls back to the instance-local #dedupe Map.

Pass { lock: false } (or lock: null) to opt out of the SAB lock on any cache — the instance-local #dedupe Map still coalesces concurrent get() calls on the same key, but sibling threads hitting the same DB file will each call valueSelector independently.

There is no cross-process coordination — two separate Node processes pointed at the same SQLite file may both run valueSelector for the same key. If you need cross-process dedup, do it at a layer above (e.g. a request-coalescing service).

When to disable the lock

• ':memory:' — already disabled for you; nothing to configure.
• Single-threaded app that doesn't spawn workers — disable to skip the cheap SAB round-trip on every get() cache miss.
• Workload where each worker uses a disjoint keyspace — the SAB adds overhead without any dedupe benefit.

File Sharding

The cache partitions its SQLite storage across N physical files (default shards: 4). Each key is hash-routed (via xxhash32) to one shard, and reads/writes for that key only touch that shard's connection. This is intended to reduce the SQLite writer-serialization ceiling: SQLite allows only one writer at a time per database file, even in WAL mode, so a single file caps concurrent-writer throughput at roughly one thread's worth of work regardless of how many threads the process spawns.

On-disk layout:

| shards | Files | | ----------- | -------------------------------------------------------------------- | | 1 | <location> (single file, legacy layout — compatible with stat()) | | N (N ≥ 2) | <location>.0, <location>.1, …, <location>.{N-1} |

With shards: 1 the cache is a single file at the given location, identical to the pre-sharding layout — useful for backup scripts or observability tools that expect one file.

Cross-instance consistency: hash routing is deterministic (xxhash32 is pure), so two Cache instances pointing at the same location with the same shards count will route the same keys to the same shards. Data persists across instance restarts.

Changing shards invalidates data. If you open a cache with shards: 2 after previously using shards: 4, keys hash-route to different shard files than they were written to. The old data is still on disk but effectively unreachable until you open with the original shard count.

Per-shard state: maxSize divides evenly across shards (maxSize / shards per shard), eviction runs per-shard on SQLITE_FULL, and stats.database.size is summed across all shards.

Batched Writes

SQLite writes are batched using setImmediate — multiple set() calls within the same microtask turn are coalesced into a single BEGIN/COMMIT transaction per shard. While a write batch is pending, the in-memory cache is corked (eviction deferred) to avoid dropping entries before they reach disk. If a per-shard batch exceeds 512 items, it is flushed immediately.

Flushes iterate shards starting at a random index on each call, so no single shard is starved when the 10 ms per-flush time budget is exhausted mid-pass — the next flush picks a different starting shard.

If a shard's database is full (SQLITE_FULL), the cache evicts the 256 oldest entries in that shard and retries up to 3 times. On other errors, the entire pending batch for that shard is dropped (items remain in the in-memory cache until natural eviction/TTL) and the error is emitted — this prevents error floods when the underlying failure is persistent (e.g. a read-only DB).

Error Handling

Cache extends EventEmitter. Non-fatal errors (SQLite failures, stale revalidation failures, background-refresh rejections from the SWR fire-and-forget path) are emitted as 'error' events when a listener is attached. If no 'error' listener is registered, errors are surfaced via process.emitWarning() instead, avoiding unhandled crashes.

Off-Peak Purge

All cache instances listen on the nxt:offPeak BroadcastChannel. When a message is received, gc() is called on every active instance, enabling coordinated cleanup during low-traffic periods.

Benchmarks

Measured on Apple M3 Pro (12 CPUs), Node 25.6.1. Throughput is ops/sec; latency is ns (median).

Single-thread hot paths

| Operation | ops/sec | p50 | p99 | | ------------------------------------------- | ------- | ------ | ------- | | get() memory hit (sequential keys) | 4.90 M | 125 ns | 666 ns | | get() memory hit (random keys) | 6.05 M | 125 ns | 584 ns | | peek() memory hit | 8.05 M | 125 ns | 250 ns | | get() memory miss, DB hit (sequential) | 303 K | 2.3 µs | 10.6 µs | | get() memory miss, DB hit (random) | 399 K | 2.3 µs | 6.4 µs | | get() cold (sync valueSelector) | 274 K | 1.8 µs | 4.5 µs | | get() memory-only hit (no DB) | 5.39 M | 125 ns | 500 ns | | get() memory-only cold (no DB) | 1.16 M | 708 ns | 2.3 µs | | get() eviction pressure (maxCount=1000) | 1.64 M | 542 ns | 1.5 µs | | delete() existing keys | 67 K | 13 µs | 44 µs | | gc() 10 K expired entries | 7.2 ms | — | — |

Shard-count comparison

Single-thread overhead of sharding on the hot paths is ≤5 % in either direction. The pay-off is under multi-thread write contention:

12 threads, partitioned cold writes (each worker writes unique keys, stressing concurrent writers):

| shards | Aggregate throughput | Scaling vs. 1 thread | Δ vs. shards: 1 | | -------- | -------------------- | -------------------- | ----------------- | | 1 | 154 K ops/s | 0.80× | baseline | | 2 | 223 K ops/s | 1.04× | +45 % | | 4 | 237 K ops/s | 1.37× | +54 % | | 8 | 182 K ops/s | 1.08× | +19 % |

12 threads, shared-keys hot-hit (mostly memory-resident; sharding shouldn't help much here):

| shards | Aggregate throughput | | -------- | -------------------- | | 1 | 7.69 M ops/s | | 2 | 7.86 M ops/s | | 4 | 7.70 M ops/s | | 8 | 7.51 M ops/s |

So shards: 4 (the default) is a large win for write-heavy multi-threaded workloads and roughly break-even otherwise. If you run single-threaded or strictly read-heavy, shards: 1 removes all sharding overhead and restores the single-file on-disk layout.

Lock option overhead (single-thread)

| Path | lock enabled | lock disabled | | --------------------------------- | ------------ | ------------- | | get() cold (sync valueSelector) | 298 K ops/s | 306 K ops/s | | get() memory hit | 7.44 M ops/s | 6.19 M ops/s |

Cross-thread SAB locking adds ~3 % on the cold path and is negligible on memory hits. It is worth leaving on unless the workload is strictly single-threaded or partitioned across workers.

Reproducing

yarn build
node scripts/bench.mjs              # full bench suite
node scripts/bench-shards.mjs       # shard-count comparison (1, 2, 4, 8 shards)

Scripts

yarn test               # run tests
yarn test:coverage      # run tests with coverage report (90%+ enforced)
yarn typecheck          # type-check without emitting
yarn build              # build for publishing