muttafa

Production-minded, low-level mutation buffering engine for JavaScript and Node.js.

muttafa is a reusable systems primitive for amortizing expensive write, persistence, and network operations. It decouples high-frequency producers from slow/expensive sinks via zero-copy generation swapping, compact storage abstractions, stationary index-based transforms, and backpressure management.

cemalloc:    reserve  ──► allocate  ──► reuse     ──► reset   (amortizes allocation cost)
muttafa:     append   ──► freeze    ──► transform ──► flush   (amortizes write & persistence cost)

What is this?

muttafa is a production-minded, low-level JavaScript/Node.js mutation buffering engine.

• ❌ This is NOT merely a batching library.
• ❌ This is NOT a database.
• ❌ This is NOT a RocksDB clone.
• ❌ This is NOT an LSM-tree implementation.

It is a reusable systems primitive for amortizing expensive mutation, write, and persistence costs.

The Mental Model

cemalloc  ──► amortizes allocation cost (reserve → allocate → reuse → reset)
muttafa   ──► amortizes mutation / write / persistence cost (append → freeze → transform → flush → release)

Problem & Solution

Instead of making expensive synchronous persistence calls on every single write:

mutation ──► expensive sink
mutation ──► expensive sink
mutation ──► expensive sink

muttafa transforms high-rate mutation streams into bounded, zero-copy, immutable batches:

mutation ─┐
mutation ─┤
mutation ─┼─► [MutableBuffer] ──(O(1) freeze)──► [ImmutableBuffer] ──► [Transform / Coalesce] ──► [Sink]
mutation ─┘

Producers continue writing to the next generation without waiting for flushes to complete (double-buffering), while strict bounded queues and backpressure guarantee that system memory never blows up.

Usable As A Foundation For

• Key-value stores & embedded databases
• Write-Ahead Log (WAL) implementations
• High-throughput filesystem & log writers
• Secondary indexing engines
• Cache persistence & snapshotting
• Real-time analytics ingestion & network batching
• Low-latency binary storage engines (in combination with memory allocators like cemalloc)

Key Capabilities

• Ultra-High Throughput: 11M–28M ops/sec with VectorStore, up to 39M ops/sec with PackedStore.
• Zero-Copy $O(1)$ Freeze: Sealing a generation takes ~20 microseconds regardless of whether it has 100 entries or 100,000 entries.
• Double Buffering by Default: Producers continue writing to Generation $B$ while Generation $A$ flushes asynchronously to the sink.
• Strict Bounded Memory & Backpressure: Configurable generation limits (maxEntries, maxBytes) and queue bounds (maxImmutableBuffers). Supports wait, reject, and custom backpressure policies.
• Memory Plateau Guarantee: Verified across 1,000,000 mutations across 100 generation cycles with 0.01 MB heap delta.
• Stationary Index Transforms: Sorts and coalesces indices (Int32Array) without shuffling or reallocating payload data.
• Sequence Barriers: Await mutation persistence up to sequence $N$ with await engine.flushThrough(seq).
• cemalloc & Arena Ready: Pluggable ArenaStore backed by ArenaAllocator (BumpAllocator or CemallocAllocator) for zero-copy binary writes and scatter/gather (writev).
• Zero Runtime Dependencies: Pure modern ESM JavaScript with complete TypeScript definitions (index.d.ts).

Quick Start

import { MutationEngine, MemorySink } from 'muttafa';

// 1. Initialize engine
const engine = new MutationEngine({
  sink: new MemorySink(),
  maxEntries: 10_000,
  maxImmutableBuffers: 2
});

// 2. High-speed synchronous appends (hot path: returns sequence number)
const seq1 = engine.put('user:101', { name: 'Ada Lovelace' });
const seq2 = engine.put('user:102', { name: 'Alan Turing' });
const seq3 = engine.delete('user:101');

// 3. Await persistence barrier
await engine.flushThrough(seq3);

// 4. Drain and close
await engine.close();

Hot-Path Non-Blocking API

To achieve peak acceptance performance without creating Promises per mutation:

import { MutationEngine, BACKPRESSURE } from 'muttafa';

const engine = new MutationEngine();

// Non-blocking tryPut
const result = engine.tryPut(key, value);

if (result === BACKPRESSURE) {
  // Backpressure encountered: wait for sink capacity asynchronously
  await engine.ready();
  engine.put(key, value);
}

Or using automated async awaiting:

await engine.putAsync(key, value);

Storage Abstractions

muttafa separates logical mutations from physical representation:

1. VectorStore (Default): Chunked parallel arrays (Float64Array, Uint8Array, Array). Zero per-mutation object allocations on append().
2. PackedStore: Structure-of-Arrays TypedArray metadata with numeric generational handles (packHandle(gen, idx)).
3. ArenaStore: Metadata in TypedArrays, raw binary payloads stored in a continuous byte arena via BumpAllocator or CemallocAllocator. Sinks consume batch.chunks (writev) or batch.materialize() with zero intermediate copying.
4. ObjectStore: Simple reference implementation storing plain JS objects for validation and oracles.

Documentation

• Architecture & Layered Design
• Lifecycle & State Machine
• Backpressure & Flow Control
• Sinks & Maybe-Promise Protocol
• Memory Model & Storage Layout
• cemalloc Integration & Roadmap
• Benchmark Analysis & 20 Questions
• Benchmark Suite Documentation

Running Tests & Benchmarks

# Run all unit, integration, and stress tests
npm test

# Run complete statistical benchmark suite
npm run bench

# Run 1M mutation memory plateau soak test
npm run bench:plateau

License

MIT