DoubleSync

Tree snapshot and delta sync in JavaScript. Content-defined chunking splits files into variable-size pieces whose boundaries are determined by content, not position — so inserting a byte at the start of a file only invalidates the chunks immediately around the insertion. Chunks are stored by SHA-256 hash, giving automatic cross-file deduplication. Folder trees are fingerprinted as Merkle trees, so unchanged subtrees can be skipped with a single hash compare.

Zero dependencies beyond @noble/hashes.

Installation

pnpm add @fizzyflow/doublesync

Quick Start

import DoubleSync, {
    CDCStore,
    DoubleSyncMemoryFolder,
    DoubleSyncMemoryFile,
    DoubleSyncSession,
} from '@fizzyflow/doublesync';

const sync = new DoubleSync({ avgSize: 8192 });

// --- Sender ---
const root = new DoubleSyncMemoryFolder('project');
const src = await root.addFolder('src');
await src.addFile('main.js', new TextEncoder().encode('console.log("hello")'));

const senderStore = new CDCStore();
const sender = new DoubleSyncSession({ sync, senderStore });

const patch = await sender.snapshot(root);  // full patch (first call)

// --- Receiver ---
const dest = new DoubleSyncMemoryFolder('copy');
const receiverStore = new CDCStore();
const receiver = new DoubleSyncSession({ sync, store: receiverStore, dest });

await receiver.apply(patch);
// dest now has src/main.js with the same content

// --- Incremental update ---
await src.addFile('main.js', new TextEncoder().encode('console.log("updated")'));

const diffPatch = await sender.snapshot(root);  // diff patch (subsequent calls)
await receiver.apply(diffPatch);
// only changed chunks are transferred

How It Works

Content-Defined Chunking (FastCDC)

Files are split into variable-size chunks using the FastCDC algorithm with normalized chunking. A rolling Gear hash scans byte-by-byte and declares boundaries when the hash's low bits match a mask. Two masks keep chunk sizes tightly centered around avgSize:

• Strict mask in [minSize, avgSize) — fewer boundaries, avoids tiny chunks
• Lenient mask in [avgSize, maxSize) — more boundaries, avoids huge chunks
• Hard clamps at minSize (default: avgSize / 4, min 64) and maxSize (default: avgSize * 8)

Because boundaries are determined by content, inserting or deleting bytes only affects the immediately surrounding chunks — all other chunks keep their identity (same hash) and don't need retransmission.

Snapshots

A snapshot is a binary document describing a folder tree at a point in time. Each file entry carries a CDC manifest (ordered list of chunk hashes) and a fingerprint (SHA-256 of content). Each folder carries a fingerprint computed recursively from its sorted children, forming a Merkle tree.

Patches

A full patch bundles a snapshot with every chunk it references — fully self-contained.

A diff patch carries only the operations needed to go from the previous snapshot to the current one (add file, replace file, remove, add empty folder), plus only the new chunks. The receiver must hold the previous snapshot to apply it.

DoubleSyncSession manages this automatically: first call to snapshot() returns a full patch, subsequent calls return diff patches.

Compression

Any patch can be wrapped in a gzip compression envelope. The receiver auto-detects and unwraps transparently. Enable with compress: 'gzip' on DoubleSyncSession.

API

DoubleSync

The core sync engine.

const sync = new DoubleSync({ avgSize: 8192, enableMerkleSkip: true });

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | avgSize | number | 8192 | Target average chunk size (must be power of 2) | | minSize | number | avgSize / 4 | Minimum chunk size (min 64) | | maxSize | number | avgSize * 8 | Maximum chunk size | | enableMerkleSkip | boolean | true | Skip unchanged subtrees by fingerprint comparison |

Methods

| Method | Returns | Description | |--------|---------|-------------| | buildSnapshot({ root, store }) | Promise<Uint8Array> | Walk tree, chunk files into store, return snapshot document | | buildPatch({ root, senderStore, receiverStore? }) | Promise<Uint8Array> | Build self-contained patch (snapshot + all chunks) | | buildDiffPatch({ root, senderStore, receiverStore?, prevSnapshot }) | Promise<{ patch, newSnapshot }> | Build incremental diff patch against previous snapshot | | applySnapshot({ snapshot, store, dest }) | Promise<void> | Reconstruct tree from snapshot + chunks in store | | applyPatch({ patch, store, dest }) | Promise<void> | Apply full patch to destination | | applyDiffPatch({ patch, store, dest, prevSnapshot }) | Promise<Uint8Array> | Apply diff patch; returns new snapshot | | missingChunks({ snapshot, store }) | Array<{ hash, length }> | List chunks referenced by snapshot that store lacks |

DoubleSyncSession

Stateful sender/receiver wrapper that tracks the previous snapshot automatically.

// Sender
const sender = new DoubleSyncSession({ sync, senderStore, compress: 'gzip' });
const patch = await sender.snapshot(root);

// Receiver
const receiver = new DoubleSyncSession({ sync, store, dest });
await receiver.apply(patch);

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | sync | DoubleSync | yes | Sync engine instance | | senderStore | CDCStore | for snapshot() | Chunk store for building patches | | store | CDCStore | for apply() | Chunk store for receiving | | dest | DoubleSyncFolder | for apply() | Target folder to write into | | compress | 'gzip' | false | no | Compress outgoing patches |

| Method | Returns | Description | |--------|---------|-------------| | snapshot(root) | Promise<Uint8Array> | Build next patch (full on first call, diff on subsequent) | | apply(patch) | Promise<void> | Apply patch (auto-detects full vs diff, handles decompression) |

| Property | Type | Description | |----------|------|-------------| | lastSnapshot | Uint8Array \| null | Snapshot from the last snapshot() or apply() call |

CDCSync

Low-level chunking engine for a single byte buffer (no tree awareness).

const cdc = new CDCSync({ avgSize: 8192 });
const { manifest, chunks } = cdc.split(data);
const rebuilt = cdc.reconstruct({ manifest, store });

| Method | Returns | Description | |--------|---------|-------------| | split(data) | { manifest, chunks } | Split buffer, return manifest + chunk list | | splitInto({ data, store }) | Uint8Array | Split + insert chunks into store; return manifest bytes | | reconstruct({ manifest, store }) | Uint8Array | Reassemble buffer from manifest + store | | missingChunks({ manifest, store }) | Array<{ hash, length }> | Chunks manifest references that store lacks |

CDCStore

In-memory content-addressable chunk store.

const store = new CDCStore();
const hash = store.put(bytes);         // store by content, returns SHA-256
store.putWithHash(hash, bytes);        // store by known hash
store.has(hash);                       // boolean
const data = store.get(hash);          // Uint8Array | null
store.delete(hash);                    // boolean

| Static Method | Returns | Description | |---------------|---------|-------------| | CDCStore.hashOf(bytes) | Uint8Array | SHA-256 of bytes | | CDCStore.hashToHex(hash) | string | 32-byte hash to lowercase hex | | CDCStore.hashFromHex(hex) | Uint8Array | Hex string to 32-byte hash |

Filesystem Abstraction

DoubleSync operates on an abstract filesystem interface. Two in-memory implementations are provided; extend DoubleSyncFile / DoubleSyncFolder to back them with real FS, S3, a database, etc.

DoubleSyncFolder (abstract)

| Method | Returns | Description | |--------|---------|-------------| | list() | Promise<Array> | Direct children (files and folders) | | walk(prefix?) | async generator | Depth-first { path, file } for all files | | findByPath(parts) | Promise<File \| Folder \| null> | Navigate by path segments | | addFile(name, bytes) | Promise<DoubleSyncFile> | Create or replace file | | addFolder(name) | Promise<DoubleSyncFolder> | Create or return existing subfolder | | removeChild(name) | Promise<void> | Remove direct child | | ensurePath(parts) | Promise<DoubleSyncFolder> | Create nested folder path |

DoubleSyncFile (abstract)

| Method | Returns | Description | |--------|---------|-------------| | getContent() | Promise<Uint8Array> | Read file bytes | | getSize() | Promise<number> | File size | | getFingerprint() | Promise<Uint8Array> | SHA-256 of content | | setContent(bytes) | Promise<void> | Write file bytes |

Concrete implementations

• DoubleSyncMemoryFolder — children stored in a Map
• DoubleSyncMemoryFile — content stored as Uint8Array

Format Detection

import { DoubleSyncFormat, MAGICS } from '@fizzyflow/doublesync';

DoubleSyncFormat.detect(bytes);        // 'manifest' | 'snapshot' | 'patch' | 'diff-patch' | 'compressed' | null
DoubleSyncFormat.isDoubleSync(bytes);  // true if any known format
DoubleSyncFormat.isPatch(bytes);       // true if full patch
DoubleSyncFormat.isDiffPatch(bytes);   // true if diff patch
DoubleSyncFormat.isCompressed(bytes);  // true if compression envelope

Patch Documents

For advanced use, parse patch documents directly:

import { DoubleSyncPatch, DoubleSyncDiffPatch, DoubleSyncSnapshot } from '@fizzyflow/doublesync';

// Full patch
const p = new DoubleSyncPatch(bytes);
p.snapshot;    // Uint8Array — embedded snapshot
p.chunkCount;  // number
for (const { hash, bytes } of p.chunks()) { /* ... */ }

// Diff patch
const d = new DoubleSyncDiffPatch(bytes);
d.prevSnapshotHash;  // Uint8Array (32 bytes)
d.opCount;           // number
for (const { kind, path, manifest, fingerprint } of d.ops()) { /* ... */ }
// kind: 'replace_file' | 'add_file' | 'add_empty_folder' | 'remove'

// Snapshot
const s = new DoubleSyncSnapshot(bytes);
s.fileCount;     // number
s.folderCount;   // number
s.totalBytes;    // bigint
for (const file of s.files()) { /* ... */ }

Wire Format

All documents use little-endian byte order.

| Document | Magic | Description | |----------|-------|-------------| | CDC Manifest | 0xCDCAA001 | Ordered chunk hash + length list for one file | | Snapshot | 0xCDCB0001 | Full tree structure with manifests and fingerprints | | Full Patch | 0xCDCB0101 | Snapshot + all referenced chunks | | Diff Patch | 0xCDCB0102 | Prev-snapshot hash + ops + new chunks only | | Compressed | 0xCDCB02FF | Gzip envelope wrapping any of the above |

Testing

pnpm test

Tests use vitest:

• test/cdc.test.js — FastCDC chunking, manifest round-trip, deduplication
• test/double-sync.test.js — end-to-end tree sync: build snapshot, apply, verify
• test/double-sync-diff.test.js — incremental diff patches, add/remove/replace ops
• test/double-sync-compressed.test.js — gzip compression envelope round-trip
• test/double-sync-stress.test.js — large trees, edge cases
• test/format.test.js — magic number detection and classification

License

ISC