@serve.zone/containerarchive

@serve.zone/containerarchive is a content-addressed incremental backup engine with a Rust core and TypeScript API for deduplicated, compressed, optionally encrypted snapshots of pipe-compatible JavaScript streams.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Container workloads do not only need file copies. They need repeatable point-in-time snapshots, low storage amplification, safe restores, and integrity checks that can run in automation. containerarchive packages those primitives behind a small TypeScript interface while leaving chunking, hashing, pack I/O, encryption, and repair work to Rust.

Highlights

• 📦 Immutable snapshot manifests with tags and multi-item backup support
• 📌 Durable snapshot pins that retention never removes
• 🧩 FastCDC content-defined chunking with SHA-256 content addressing
• ♻️ Cross-snapshot deduplication through a global chunk index
• 🧬 Crash-safe index generations published through a redundant, atomically exchanged head
• 🧹 Retention pruning with whole-pack garbage collection that never deletes before the index stops referencing
• 🗜️ gzip by default, zstd selectable at init()
• 🔐 Optional AES-256-GCM encryption with Argon2id-derived passphrase wrapping
• 🧱 8 MB target pack files with sidecar .idx lookup data
• 🔍 Quick, standard, and full repository verification modes
• 🔒 OS-enforced descriptor locks retained across each repository mutation
• 🧯 Bounded, fail-closed parsing and restore preflight checks for repository-controlled metadata

Current Safety Gates

This release deliberately refuses maintenance operations that cannot yet meet the index-generation publication contract:

• deleteSnapshot() durably removes one restore-point manifest under the repository lock and is idempotent for lost-response retries. Shared pack data is reclaimed by the next prune().
• repair() and reindex() reject before mutation. A verified rebuild of the index generation from pack index files is required before they can be enabled safely.
• Reed-Solomon parity configuration remains part of the repository schema, but parity generation and recovery are disabled until groups persist across ingests and protect both .pack and .idx data. prune() deletes any existing parity group that protects a deleted pack.

These gates prevent partial cleanup, stale-index resurrection, and repair of an unopenable repository through an unsafe normal-open path.

Index Generations and Migration

The global index is a sequence of immutable generations. A generation document lists the content-addressed index segments whose union is the index, and a redundant two-slot head (generations/heads/HEAD.A, HEAD.B) names the current generation together with its exact parent. Ingest, prune() and closure import write their new segments and document first and then advance the head by atomically exchanging its older slot (RENAME_EXCHANGE), so a crash leaves either the previous or the new generation current, never a mixture. Every locked operation first recovers interrupted head staging and removes every generation object that neither the current nor the previous generation reaches. Readers that do not take the lock (open(), restore(), verify()) read the current generation; a read that races a concurrent commit is retried, and fails with Index generation kept changing while it was read if the head keeps moving. A generation that would list more than 64 segments is written compacted, so the segment count stays bounded. The head binds the exact bytes of config.json: a repository whose config was edited after initialization fails to open.

Repositories written by releases up to 0.6.x use the legacy index/ layout. Every operation except the static helpers below refuses them with Repository migration is required: ...:

const { format } = await ContainerArchive.inspect('/var/backups/app');
// 'indexGenerations' | 'legacyIndex' | 'migrationInProgress'

if (format !== 'indexGenerations') {
  const result = await ContainerArchive.migrate('/var/backups/app');
  console.log(result.previousFormat, result.indexedChunks, result.indexSegments);
}

migrate() needs no passphrase and runs under the repository lock. It first renames index/ to index.migrating/, then writes the genesis generation and the head, then removes index.migrating/; a crash at any step is completed by calling migrate() again, and a migrated repository is left unchanged. Migration is one-way: from its first step, releases up to 0.6.x fail closed with Index directory is missing; repository repair is required instead of reading a stale index. Existing repository roots keep the modes they were accepted with; the new generations/ directories are created with mode 0700 and their files with 0600.

migrate(source, { destination }) leaves the source byte-identical, so the release that wrote it can still read it (a rollback copy), and writes the converted repository to destination instead:

await ContainerArchive.migrate('/var/backups/app', { destination: '/var/backups/app-converted' });

It takes the source's existing operating-system lock without writing lock metadata, so no writer changes the source during the copy, and refuses a source without locks/repository.lock rather than creating one. It copies config.json and every file of packs/, snapshots/, keys/ and pins/ into <destination>.migrating, each through a fixed stage file that is synced before it is renamed into place, writes the genesis generation there from the source's legacy index, and renames the staging root to destination as its commit point. Every file is a copy with its own inode, because repository files must have exactly one hard link. The destination must not exist and must lie outside the source, on the source's filesystem; another filesystem is refused with unsupported-filesystem. Before copying, the free space of that filesystem must cover the bytes still to copy plus a reserve of 64 MiB and 1 % of those bytes, or the call rejects with ContainerArchiveReserveReachedError (repository-reserve-reached); a staging root this call created is removed again, while the staging root of an interrupted copy is kept for the next attempt. A killed or cancelled copy resumes on the next identical call: stage residue is removed, files already staged are compared with the source (data-corruption on a mismatch) and the rest is copied. A staging root holding anything a migration does not write, or another repository's config, is refused. Calling it again after completion replays the result without touching either repository. A staged or restored repository often lacks locks/repository.lock, because replication leaves runtime state out. Declare such a directory with sourceIsPrivateCopy: true (together with destination) when no other process opens it: containerarchive then locks the source root directory itself against a second migration instead of requiring the lock file, holds an existing lock file as well, and keeps every other guarantee above: the source stays byte-identical, the copy resumes, and another filesystem or too little space is refused by name. It cannot exclude a release that would lock the repository through its lock file, so never declare a repository something else may use. Only legacy-layout sources are accepted: a current-format source needs no migration, and an interrupted in-place migration is completed with migrate(source).

A copy can take long, so it reports progress, can be estimated beforehand and can be stopped:

const source = '/var/backups/app';
const destination = '/var/backups/app-converted';
const estimate = await ContainerArchive.estimateMigration(source, { destination });
if (!estimate.fits) {
  throw new Error(`needs ${estimate.bytesRequired} free bytes, has ${estimate.bytesAvailable}`);
}
const controller = new AbortController();
await ContainerArchive.migrate(source, {
  destination,
  signal: controller.signal,
  onProgress: (progress) => {
    if (progress.phase === 'copy') {
      console.log(`copying: ${progress.filesCopied} of ${progress.filesTotal} files`);
    } else {
      console.log(`indexing: ${progress.indexSegmentsLoaded} of ${progress.indexSegmentsTotal} legacy segments read`);
    }
  },
});

estimateMigration() takes the same source hold and runs the same checks as the copy up to its space check, then reports filesTotal and bytesTotal (every file the copy covers), bytesToCopy (fewer when an interrupted copy resumes), bytesRequired (bytesToCopy plus the reserve above), bytesAvailable and fits, without creating or changing anything. Both use one space rule, so migrate() rejects with ContainerArchiveReserveReachedError exactly when fits is false; for a completed destination, or a staging root whose copy has finished, nothing remains to copy and bytesRequired is 0. It refuses what the copy refuses up front, for example a source a writer holds or a destination on another filesystem.

onProgress (with a destination only) receives IMigrationProgress: { operation: 'migrate', phase, filesCopied, filesTotal, bytesCopied, bytesTotal, indexSegmentsLoaded, indexSegmentsTotal, indexSegmentsWritten, indexSegmentsVerified }, in two phases. In phase: 'copy' the file and byte counters advance: the totals are fixed before the first file is copied; a file counts once it is in the staging root, copied now or, when an interrupted copy resumes, compared with the source, and bytes count per copied or compared MiB. Then phase: 'index' finishes the migration, and its counters advance once per index segment: indexSegmentsLoaded of indexSegmentsTotal legacy segments read and validated, indexSegmentsWritten segments of the genesis generation written, and indexSegmentsVerified segments of that generation read back before the staging root is renamed into place. This work grows with the index, so it reports like the copy. Between two reports runs at most one segment (128 MiB at most) or one bounded set of fixed metadata steps: the staging root's fsync (every copied directory was synced as it completed), the staging repository's capability probe and the in-memory sort of at most 1,000,000 index entries before the first genesis segment, writing the genesis document and head with their fsyncs and reopening the written generation before the read-back, or the final rename. None of these grows with the number of files or bytes in the archive. A resumed migration whose genesis generation is already written goes straight to phase: 'index' with the file counters at 0 and indexSegmentsTotal 0, and reports each segment it reads back. The phase only moves from copy to index, and no counter shrinks. The engine checks the counters once per second and reports only when they advanced, so there is at most one call per second. A migration that stalls, for example in a filesystem call that does not return, reports nothing. The last call arrives before migrate() resolves; it is in phase: 'index', with filesCopied === filesTotal, bytesCopied === bytesTotal and indexSegmentsLoaded === indexSegmentsTotal. A consumer that ignores phase still sees every call. A replayed destination reports nothing. onProgress must not throw; a throw surfaces as an unhandled error and does not stop the migration. The bridge applies its five-minute limit to inactivity, not to the whole migration.

signal stops the copy before its next file, or before it writes the genesis generation, leaving <destination>.migrating exactly as a crash there would; the call rejects with ContainerArchiveCancelledError and the next identical call resumes. Past that point the migration completes and returns its result. An in-place migration stops only before it fences the legacy index (see Cancellation).

Before calling migrate(), stop every process that holds the repository open with an older release, or upgrade it to this release. Migration waits for the repository lock, so it never interleaves with an older release's write; afterwards every locked operation of releases 0.2.0 to 0.6.x reloads the index, finds index/ missing, and fails closed before it writes anything. An older handle that stays open can, however, still serve lock-free restores from the index it loaded before the migration, and those fail once prune() has deleted packs they would read.

inspect() and every other operation fail with Data corruption: Repository holds a legacy index beside index generations or an interrupted migration when an index/ directory exists beside generations/heads/. Releases 0.2.0 and later cannot create that state after migration; it arises only when index/ is restored by hand or written by a pre-0.2.0 release, which takes no operating-system lock. To recover: stop every process using the repository; move index/ out of the repository and keep it as evidence; open the repository with this release and run prune({}, true). If the dry run succeeds, no snapshot depends on the moved segments. If it names a snapshot that references a chunk missing from the global index, that snapshot was written by the older process after migration and its chunks are not in any index generation: remove it with deleteSnapshot() and take a new backup, because reindex(), which could rebuild the index from pack index files, is not available yet.

Install

pnpm add @serve.zone/containerarchive

ContainerArchive ships statically linked musl binaries for Linux x64 and arm64 (dist_rust/containerarchive_linux_amd64_musl and dist_rust/containerarchive_linux_arm64_musl); they run on glibc and musl distributions alike, and other platforms fail at construction instead of searching for another binary. An application that installs the engine elsewhere, for example beside its own compiled executable, passes binaryPath to init(), open(), inspect(), migrate(), estimateMigration() or importClosure(): that exact file is the only candidate, the packaged binary is not used, and a missing, non-file or non-executable path rejects with RustBinaryLocatorError (code ERR_RUST_BINARY_EXPLICIT_PATH_INVALID) without starting any engine or changing the file's permissions. An empty binaryPath throws. The binaries allocate through mimalloc instead of musl's allocator and compress and inflate gzip chunks with zlib-rs. The package carries the third-party license notices for everything compiled into these binaries (Rust crates, the zstd and mimalloc C libraries, the Rust standard library, musl and the GCC runtime) in native-notices/. It requires Linux filesystem support for Unix-domain sockets, flock, descriptor-relative no-follow traversal, no-replace/exchange renames, and durable file and directory synchronization; initialization and open fail closed when required capabilities are unavailable.

Architecture

The TypeScript class manages the developer-facing API and uses @push.rocks/smartrust to control the compiled Rust binary. Large data does not travel through JSON IPC; the TypeScript side opens temporary Unix sockets and streams bytes directly to or from Rust.

Node.js app
  |
  | TypeScript API: ContainerArchive
  |
  | JSON IPC for commands, Unix sockets for data streams
  v
Rust engine
  |
  | chunk -> hash -> compress -> encrypt -> pack -> snapshot
  v
repository directory

Filesystem and Stream Security

Every mutation in the current repository format, including repository initialization, acquires locks/repository.lock with an operating-system flock and retains the verified repository-root descriptor for the full operation. During a locked mutation, config/index state, capacity checks, pack shards, index segments, and snapshot manifests are resolved from that exact descriptor. Renaming or replacing the original repository path therefore cannot redirect a locked publication. Immutable config, key, pack, index, and snapshot files use no-replace publication; an existing name is accepted only when its complete bytes match exactly. Encrypted key material is loaded during open() and cached in memory rather than re-resolved through the mutation lock.

Initialization creates only the final repository path component. Its parent must already exist and either belong to the current user without group/world write permission or be a root-owned sticky directory such as /tmp. A supplied empty destination must already be a current-owner 0700 directory. Initialization rejects symlink components and nonempty destinations without creating locks/; it retains the exact parent/root descriptors, takes an auxiliary root-directory flock, then holds the authoritative locks/repository.lock flock until config.json has been written last and the root has been synced and revalidated.

After the requested repository path is resolved, the completed repository root must belong to the current user and must not be group/world writable. During locked mutations, descriptor-relative child directories used for config/index refresh and publication receive the same ownership and write-permission checks, do not follow symlinks, and must remain on the root filesystem. Path-based read-only helpers canonicalize requested children and constrain them to their expected parent directories, but they do not provide the same retained-descriptor guarantee. Existing completed 0755 roots and descriptor-relative mutation directories are accepted when opened, while new initialization directories are exact 0700. The authoritative lock file is stricter: it must be a current-owner, single-link regular file with exact mode 0600. New writes create it with that mode and bind its metadata before truncation.

Repositories initialized by version 0.1.4 normally do not contain the fixed repository.lock file. If an existing or externally created lock file has mode 0644 (or otherwise fails validation), this release rejects it without modifying or automatically repairing it. Stop every repository process, verify the file and repository ownership, then perform any operator-directed remediation; do not change or remove a lock file while a process may still hold it.

Each ingest/restore stream uses a cryptographically random 0700 temporary directory, a current-owner 0600 Unix socket, and a separate 32-byte random capability. The resolved temporary-directory chain must consist only of root/current-owner non-writable ancestors or a root-owned sticky directory such as /tmp; endpoint directory and socket device/inode identities are retained and revalidated before Rust receives the path and again during cleanup. Peers must send the fixed-size capability frame before data transfer; comparison is constant-time. Incorrect peers are disconnected while the listener remains available for the legitimate Rust peer. Authentication is limited to five seconds with at most eight provisional peers, and cleanup never recursively deletes an unexpected path.

After authentication, ingest uses bounded data frames followed by an explicit completion frame. Socket EOF without that completion is an error, so a Node.js source that emits bytes and then raises an error cannot be acknowledged as a shorter successful snapshot. ingestMulti() installs listeners for every source before waiting for any socket; an error from a later item therefore fails the whole ingest even while Rust is still reading an earlier item. Before the head names the index generation that references its packs, failure cleanup removes only pack and .idx inodes created by that ingest and never removes an exact replay of a pre-existing immutable object. If that generation is already committed and snapshot publication then fails, the indexed packs are retained as deduplication data until a prune() finds them unreferenced.

Ingest and closure-import sources implement IContainerArchiveInputStream: on(), off(), pipe(), and destroy() are required, while unpipe() is optional. This supports Node.js Readable streams and streamx-compatible producers while retaining deterministic error cleanup.

Quick Start

import { createReadStream, createWriteStream } from 'node:fs';
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.init('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshot = await repo.ingest(createReadStream('/tmp/database.sql'), {
  inactivityTimeoutMs: 5 * 60 * 1000,
  tags: {
    service: 'postgres',
    environment: 'production',
  },
  items: [{ name: 'database.sql', type: 'database-dump' }],
});

console.log(snapshot.id, snapshot.newChunks, snapshot.reusedChunks);

const restored = await repo.restore(snapshot.id, { item: 'database.sql' });
restored.pipe(createWriteStream('/tmp/restored-database.sql'));

await repo.close();

Open an Existing Repository

import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.open('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshots = await repo.listSnapshots({
  tags: { service: 'postgres' },
});

Repositories initialized without a passphrase are unencrypted. Encrypted repositories require the passphrase on open().

init() accepts a new-repository passphrase, partial FastCDC chunking overrides, compression (gzip, the default, or zstd), and packTargetSize. Passphrases must contain 1 to 1024 UTF-8 bytes. Chunk sizes must be positive safe integers with minSize <= avgSize <= maxSize <= 64 MiB; the only supported algorithm is fastcdc. packTargetSize must be a positive safe integer no larger than 1 GiB. New repositories omit parity configuration while parity publication/recovery remains safety-gated, so every individually valid chunk/pack combination has the documented init contract. TypeScript validates these values before starting the Rust process, and Rust independently validates the strict IPC shape and merged configuration before creating the destination.

Multi-Item Snapshots

Use ingestMulti() when a single restore point needs several streams, for example a DB dump plus a config archive.

import { createReadStream } from 'node:fs';

const snapshot = await repo.ingestMulti([
  {
    name: 'database.sql',
    type: 'database-dump',
    stream: createReadStream('/tmp/database.sql'),
  },
  {
    name: 'volumes.tar',
    type: 'volume-tar',
    stream: createReadStream('/tmp/volumes.tar'),
  },
], {
  tags: { service: 'nextcloud', kind: 'full-backup' },
});

console.log(snapshot.items.map((item) => item.name));

Listing, Filtering, and Restore

const allSnapshots = await repo.listSnapshots();

const recentProductionSnapshots = await repo.listSnapshots({
  tags: { environment: 'production' },
  after: '2026-05-01T00:00:00Z',
});

const snapshot = await repo.getSnapshot(recentProductionSnapshots[0].id);

const stream = await repo.restore(snapshot.id, {
  item: snapshot.items[0].name,
});

listSnapshots() returns ISnapshotSummary entries, newest first: id, version, createdAt, tags, originalSize, storedSize, chunkCount, newChunks, reusedChunks, itemCount, and pin. Summaries never carry item or chunk lists; getSnapshot(id) returns the full ISnapshot manifest of one snapshot. A filter matches when every given tag matches exactly and the creation time lies within the inclusive after/before bounds, which must be RFC 3339 timestamps and are compared as instants. A malformed filter or an unknown filter field is refused rather than ignored.

Verification

const quick = await repo.verify({ level: 'quick' });
const full = await repo.verify({ level: 'full' });

if (!full.ok) {
  console.error(full.errors);
}

Verification levels are intentionally different tradeoffs: quick checks index consistency, standard reads pack metadata/checksums, and full rehydrates chunk content for the strongest validation. Verification reports observed snapshot enumeration, pack and index I/O, index parsing, and completed chunk checks as management activity, so a progressing full verification can run longer than five minutes while five minutes without verification progress still fails closed.

repair() and reindex() are currently safety-gated as described above. unlock() cannot break a live operating-system lock; process exit releases lock ownership automatically.

Retention Pruning

const preview = await repo.prune({ keepLast: 7, keepDays: 30 }, true);
console.log('would free bytes', preview.freedBytes);

for (const decision of preview.snapshots) {
  console.log(decision.snapshotId, decision.action, decision.reasons);
}

const result = await repo.prune({ keepLast: 7, keepDays: 30 });
console.log(result.removedSnapshots, result.removedPacks, result.freedBytes);

A destructive prune runs under the repository lock: it removes the manifests retention does not keep, commits the next index generation without the entries of every pack that holds no chunk of a remaining snapshot, and only then deletes those packs (.idx before .pack), every parity group that protects one of them, and every unreachable generation object. Packs are deleted whole; a pack that still holds one referenced chunk is kept entire. A crash at any point leaves every remaining snapshot restorable, and the next prune() completes the cleanup. A dry run reports the same decisions, packs and bytes without changing anything.

Retention is repository-wide: keepDays, keepWeeks and keepMonths keep one snapshot per calendar day, week or month across the whole repository, whatever its tags. A consumer that keeps several independent groups in one repository (for example one backup series per service) must apply its own per-group retention with deleteSnapshot() and call prune({}, false): without a retention policy prune keeps every snapshot and only collects garbage.

The result reports removedSnapshots, keptSnapshots, and one decision per snapshot, newest first. A kept snapshot lists every rule that keeps it: pinned, keepLast, keepDays, keepWeeks, keepMonths, or noRetentionPolicy when no rule is set. A removed snapshot has no reasons. Packs referenced by a kept snapshot, pinned ones included, are never counted as reclaimable.

Snapshot Pins

A pin is an explicit retention hold with a reason. Retention never selects a pinned snapshot, and deleteSnapshot() refuses one until it is unpinned. Pinned snapshots take no part in the retention policy: they neither fill a keepLast slot nor occupy a day, week, or month bucket, so the remaining snapshots are retained exactly as they would be without the pinned ones.

import { createReadStream } from 'node:fs';
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.open('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

// Pin at ingest: the snapshot is never observable unpinned.
const archived = await repo.ingest(createReadStream('/var/lib/legacy/export.tar'), {
  items: [{ name: 'export.tar', type: 'volume-tar' }],
  pin: { reason: 'only copy of the retired export volume' },
});
console.log(archived.pin); // { snapshotId, reason, pinnedAt }

// Pin or unpin an existing snapshot.
const [latest] = await repo.listSnapshots({ tags: { service: 'postgres' } });
await repo.pinSnapshot(latest.id, 'legal hold');
await repo.unpinSnapshot(latest.id); // { unpinned: true }

await repo.close();

listSnapshots(), getSnapshot(), ingest(), and ingestMulti() return every snapshot with pin set to its pin or null. The pin reason must be non-blank, at most 4096 UTF-8 bytes, and free of control characters. Repeating a pin with the same reason returns the existing pin, so a lost response can be retried; a different reason is refused until the snapshot is unpinned. unpinSnapshot() is idempotent and also clears a pin whose snapshot no longer exists.

Snapshot manifests stay immutable. A pin is its own immutable record, pins/<snapshotId>.json, published with the same no-replace, synced publication as manifests under the repository lock, and unpinning removes that exact record. An ingest with a pin publishes the pin record before the manifest and removes it again if the manifest cannot be published. The pins/ directory is created on first use, so repositories created by earlier releases open unchanged. Pin records are plaintext JSON, the same exposure as snapshot tags, even in encrypted repositories: do not put secrets in a pin reason. Pins are local retention state and are not part of closure exports; pin imported snapshots in the destination repository.

Resource Limits and Cancellation

Repository-controlled data is bounded before allocation or output. The most relevant public restore defaults are:

| Limit | Default | Supported maximum | | --- | ---: | ---: | | Chunk plaintext | 8 MiB | 64 MiB | | Selected item | 1 GiB | 1 TiB | | Selected total output | 1 GiB | 1 TiB | | Referenced stored bytes | 2 GiB | 2 TiB | | Chunk references | 100,000 | 1,000,000 |

Pass restore(snapshotId, { limits: { ... } }) to lower or raise these limits. Both ingest()/ingestMulti() and restore() accept inactivityTimeoutMs; it must be a positive safe integer, and the default and supported maximum are five minutes. Closure export options likewise accept inactivityTimeoutMs plus limits.maxTotalBytes and limits.maxObjects. Import adds limits.maxVerifiedPlaintextBytes, which bounds the unique plaintext decrypted, decompressed, and hashed before publication. Import defaults are 64 GiB, 100,000 objects, and 64 GiB of verified plaintext; callers can raise them up to 2 TiB, 400,506 objects, and 1 TiB respectively. The ingest timer starts only after the legitimate Rust peer authenticates, so a later multi-item socket does not expire while Rust is still processing an earlier item. Long-running bridge commands use the same five-minute inactivity boundary. Authenticated ingest-frame consumption, successful restore or closure socket I/O, closure preflight and validation progress, and completed durable publication phases reset it; elapsed wall-clock time alone does not generate liveness. Destroying a returned restore or closure-export stream cancels its Unix socket so the serial Rust management loop can continue. Closing, aborting, or erroring an import source before framed completion cancels the import; after transfer, the authenticated socket remains open so peer loss cancels validation before publication starts.

ingest() requires exactly one metadata item. ingestMulti() requires between one and 64 items, and one ContainerArchive instance admits at most 64 aggregate live ingest items across concurrent calls. Admission limits are checked before private socket directories or listeners are allocated.

Free-Space Reserve

ingest(), ingestMulti() and ContainerArchive.importClosure() accept reserveFreeBytes, a positive safe integer: the bytes that must stay free on the repository filesystem. The Rust process reads the space available to an unprivileged writer (fstatvfs, f_bavail × f_frsize) on the repository root before every write that grows the repository and refuses the write unless its bytes plus the reserve are available:

• Before every pack write, an ingest needs the .pack and .idx bytes, the parity of the group the pack completes, and the bound of the index commit that can follow it (a full batch of 16,384 entries, or the whole index when that commit compacts the generation).
• Before the snapshot manifest, an ingest needs the serialized manifest, plus the pin record when the snapshot is published pinned.
• A closure import needs the receipt's payload bytes before it writes anything, each object's bytes before writing it, and the bound of the genesis index generation plus config.json before publication.

These are byte bounds; filesystem block rounding and metadata come out of the reserve. A refusal rejects the call with ContainerArchiveReserveReachedError (code 'repository-reserve-reached'), which carries the refused write as operation ('pack write', 'snapshot manifest', 'closure import', …) and availableBytes, neededBytes and reserveBytes:

import { ContainerArchiveReserveReachedError } from '@serve.zone/containerarchive';

try {
  await repo.ingest(source, { reserveFreeBytes: 4 * 1024 ** 3 });
} catch (error) {
  if (error instanceof ContainerArchiveReserveReachedError) {
    console.warn(`${error.operation} needs ${error.neededBytes} bytes, ${error.availableBytes} available`);
  }
  throw error;
}

A refused ingest aborts through its ordinary failure path: no snapshot manifest or pin is written, and every pack it wrote but had not indexed yet is removed. An ingest commits its index in batches, so the packs of batches committed before the refusal stay behind as unreferenced garbage; prune({}, false) reclaims them. A refused closure import leaves disposable staging evidence like any other failed import (see Portable Snapshot Closure Export). Without reserveFreeBytes nothing is checked.

Concurrent Calls

One ContainerArchive instance drives one Rust process, and that process executes one command at a time. Calls made while another command runs wait in a first-in, first-out queue and reach Rust only when the previous command has settled. Their request deadline, their streaming inactivity deadline and, for restore() and exportClosure(), their socket inactivity timer start at that moment, so a listSnapshots(), deleteSnapshot() or restore() issued while a long prune(), verify() or ingest runs waits for it instead of timing out while Rust has not yet read it. A command whose socket stream Rust is still serving keeps the queue until its Rust command settles: an unread restore() stream blocks later commands until its inactivity timeout cancels it.

At most 64 commands wait; the next one is refused with ContainerArchive command queue is full: <command> was refused because 64 commands already wait behind <active command>. Ingest item admission is counted separately and includes waiting ingests. A waiting ingest() or ingestMulti() observes its input streams: a source that errors, closes before its end, or aborts removes the call from the queue, releases its item admission, and rejects it with <command> was cancelled while it waited in the command queue: input stream <item name> failed, whose cause is the source failure. getCommandQueue() reports the active command with its running time and every waiting command with its position and waiting time:

const queue = repo.getCommandQueue();
// { active: { command: 'prune', runningMs: 81234 },
//   waiting: [{ command: 'listSnapshots', position: 1, waitingMs: 4210 }],
//   maximumWaiting: 64 }

close() rejects every waiting command with ContainerArchive was closed while <command> waited <n> ms in the command queue, refuses later calls with ContainerArchive is closed; <command> was refused, lets the active command finish, and then closes the repository and ends the Rust process. A repeated close() returns the same promise.

A command whose request deadline or streaming inactivity deadline expires is cancelled in Rust as described under Cancellation; the queue waits until Rust has stopped it.

Each snapshot manifest is limited to 32 MiB. listSnapshots() reads and filters the manifests one at a time, so the size of the backed-up data does not limit it; the encoded summaries of the matching snapshots are limited to 32 MiB, and a larger listing is refused with an error that asks for a narrower filter, never truncated. verify() loads and checks one manifest at a time, and prune() plans retention on the snapshot summaries and then collects the chunk references of the kept snapshots one manifest at a time into a set that never outgrows the global index; neither is limited by the combined size of the manifests. The in-memory global index is capped at 1,000,000 unique chunks and 512 MiB of index-segment source data.

prune() plans retention on the same summaries, so a repository whose summaries of all snapshots exceed 32 MiB is refused with "Retention planning refused"; verify() checks one manifest at a time and is bounded only by the per-manifest limit.

Snapshot item names must be non-empty and unique, one management process can own only one open repository at a time, and verification stops after 10,000 reported integrity errors instead of growing an unbounded response.

Cancellation

ingest(), ingestMulti(), restore(), verify(), exportClosure(), prune() and ContainerArchive.migrate() accept an AbortSignal as signal (prune(retention, dryRun, { signal })). The Rust process advertises cooperative cancellation: aborting the signal, or a request deadline that expires, sends Rust a cancel message for the running command, and Rust stops it at its next safe point. A safe point is a place where stopping leaves the repository exactly as a crash there would:

| Command | Safe points | Commit point | | --- | --- | --- | | ingest(), ingestMulti() | before each input frame (also while waiting on a stalled source), before each pack write, after the last pack | publishing the final index batch and the snapshot manifest | | restore() | preflight, before each chunk (also while the consumer does not read) | none; restore changes nothing | | verify() | before each snapshot, pack and chunk check | none; verify changes nothing | | exportClosure() | preflight, before each write (also while the consumer does not read) | none; export changes nothing | | prune() | after planning, after each removed snapshot, while collecting references, before and after the index commit, after each removed pack and parity group, before the generation sweep | none; every step boundary is crash-equivalent and the next prune completes the work | | ContainerArchive.migrate() in place | before the legacy index is fenced | renaming index/ to index.migrating/ | | ContainerArchive.migrate() with destination | before each file is copied, before the genesis generation is written | writing the genesis generation into <destination>.migrating |

A stopped command rejects with ContainerArchiveCancelledError (code 'cancelled'), whose operation names the command. A cancelled ingest publishes no snapshot and removes the packs it had not indexed; packs of index batches it committed earlier are garbage that prune({}, false) reclaims. A cancelled destructive prune reports prune: { removedSnapshots, indexCommitted, removedPacks }. A command already past its commit point completes, and the call returns its result: an ingest cancelled while it publishes returns the snapshot. A command still waiting in the queue, or aborted before it reached Rust, also rejects with ContainerArchiveCancelledError. A restore() or exportClosure() stream fails with the same error at once.

The command keeps the queue until Rust has answered the cancellation, so the next command never overlaps it. A command that has not stopped 30 seconds after its cancellation rejects with a ContainerArchiveError of code 'command-unsettled'; its Rust process is terminated, and the next command starts a new process and reopens the repository, which recovers as after a crash. When stdin of the Rust process closes, every running command is cancelled the same way. A command whose process exits before it answers its cancellation rejects with code 'process-exited', and the next command likewise starts a new process and reopens the repository. To reopen after such a respawn, an instance holds the passphrase it was opened or initialized with in memory while the repository is open; it is used for nothing else and close() clears it.

const controller = new AbortController();
setTimeout(() => controller.abort(), 60_000);
try {
  await repo.verify({ level: 'full', signal: controller.signal });
} catch (error) {
  if (error instanceof ContainerArchiveCancelledError) {
    // verify stopped at a safe point
  }
  throw error;
}

Errors

Every failure the Rust process reports rejects with a ContainerArchiveError, whose code is stable and equals its name, so a failure is recognisable across separately installed copies of this package; the bridge error is its cause. containerArchiveErrorCodes lists the codes. Failures with a dedicated class and structured fields:

| Class | Code | Fields | | --- | --- | --- | | ContainerArchiveReserveReachedError | repository-reserve-reached | operation, availableBytes, neededBytes, reserveBytes | | ContainerArchiveCancelledError | cancelled | operation, prune for a destructive prune | | ContainerArchiveMigrationRequiredError | migration-required | — |

Other codes include repository-locked, snapshot-pinned, not-found, data-corruption, encryption-error, resource-limit-exceeded, recovery-required, ambiguous-commit, poisoned-repository, invalid-request and repository-not-open. Validation in TypeScript still throws a plain Error before anything reaches Rust, and a transport failure such as the Rust process exiting keeps smartrust's RustBridgeRequestError.

Events

ContainerArchive#on() exposes RxJS subscriptions for progress and integrity signals. ingest:progress fires at most once per second while an ingest reads or stores data, with cumulative counts for the running command; the last one arrives before the ingest returns. A migration, which runs without an instance, reports through migrate()'s onProgress instead (see Index Generations and Migration).

const subscription = repo.on('ingest:progress', (event) => {
  console.log(event.operation, event.bytesRead, event.newChunks, event.reusedChunks, event.storedBytes, event.packsWritten);
});

repo.on('ingest:complete', (event) => {
  console.log('snapshot complete', event.snapshotId);
});

repo.on('verify:error', (event) => {
  console.error('verification error', event.pack, event.chunk, event.error);
});

subscription.unsubscribe();

Repository Layout

An initialized repository is a directory with predictable data stores.

repo/
  config.json
  packs/
    data/
    parity/
  snapshots/
  pins/
  generations/
    heads/
    documents/
    segments/
    staging/
  keys/
  locks/

| Path | Purpose | | --- | --- | | config.json | Repository ID, chunking config, compression, encryption, pack target size, and parity config. | | packs/data | Binary pack files and pack indexes. | | packs/parity | Reserved for the safety-gated parity format. | | snapshots | Immutable JSON snapshot manifests. | | pins | Immutable JSON pin records, one per pinned snapshot; created on first pin. | | generations/heads | Redundant head naming the current index generation (HEAD.A, HEAD.B). | | generations/documents | Immutable, content-addressed generation documents listing index segments. | | generations/segments | Immutable, content-addressed index segments mapping chunk hashes to pack locations. | | generations/staging | Fixed staging slots for objects being installed; residue is removed under the lock. | | keys | Wrapped encryption keys for passphrase-protected repositories. | | locks | Persistent metadata file whose open descriptor carries the operating-system lock. |

Portable Snapshot Closure Export

exportClosure() emits an opaque stream containing the selected snapshot manifests, a deterministic scoped index, encryption key files when required, and only the immutable pack/index files reachable from those snapshots. Packs are indivisible immutable objects, so a reachable pack can contain physical bytes that are not referenced by the selected manifests. Unrelated manifests, unreachable packs, and unselected entries in the scoped global index are excluded, but a copied immutable pack sidecar index can still describe unselected chunks that share a reachable pack.

import { pipeline } from 'node:stream/promises';

const exported = await repository.exportClosure([snapshot.id]);
const [, receipt] = await Promise.all([
  pipeline(exported.stream, destination),
  exported.completion,
]);

const importedReceipt = await ContainerArchive.importClosure(
  stagingRepositoryPath,
  source,
  {
    expectedReceipt: receipt,
    passphrase: process.env.ARCHIVE_PASSPHRASE,
  },
);

console.log('imported closure', importedReceipt.sha256);

Each object and the whole stream are SHA-256 integrity-hashed. The local Unix peer is capability-authenticated, but the exported artifact is not self-authenticating: bind the complete receipt — format version, snapshot IDs, object, pack, and chunk counts, payload and archive byte lengths, and digest — to an authenticated outer protocol before remote transport. The closure config disables parity because a selected closure does not contain or claim repository-wide parity coverage.

Export is fail-before-data for preflight errors and is bounded to 256 selected snapshots, 1,000,000 aggregate chunk references, 400,506 objects, 32 MiB of aggregate selected-manifest source data, and 2 TiB of encoded output. Import requires an absolute path beneath a safe existing parent and accepts only a new destination or a current-owner, exact-0700, empty destination. It creates the repository skeleton itself, validates the exact selected object set, and publishes config.json last. It has no live-merge or force mode. After staging starts, a failure before config publication can leave an invalid, config-less repository. A durability error after config publication is reported as an ambiguous commit and can leave a valid config present. Treat either outcome as disposable staging evidence and discard only the exact staging directory that the caller allocated exclusively for this import before retrying. A failure can also occur before creating the destination, and a pre-existing destination that fails the import preconditions must never be deleted as cleanup. Do not parse closures in application code, and do not open or otherwise trust a staging repository until importClosure() resolves successfully.

Export acquires the repository operation lock, refreshes the published index, and retains one verified root descriptor through preflight and streaming. It therefore cannot combine cached state with a replaced repository path. Import checks restore-equivalent chunk bounds and item sizes, and decrypts, decompresses, and content-hashes every selected unique chunk before publication. Immediately before publishing config.json, Rust sends a fixed publication marker over the authenticated socket and waits for the local peer's acknowledgement. Cancellation remains safe before that acknowledgement; after it, the short durable commit and one-shot Rust process finish even if the peer or management transport disappears.

Encrypted closures include wrapped repository key files and must be classified and protected like the source repository. Import requires the repository passphrase so every selected chunk can be decrypted and content-verified before publication; the passphrase is also required when the imported repository is opened. Supplying a passphrase for an unencrypted closure is rejected.

API Surface

| API | Purpose | | --- | --- | | ContainerArchive.init(path, options?) | Create a new repository and return an open instance. | | ContainerArchive.open(path, options?) | Open an existing repository in the index-generation format. | | ContainerArchive.inspect(path, options?) | Report a repository's index format without opening or changing it. | | ContainerArchive.migrate(path, options?) | Convert a legacy-layout repository to index generations, or complete an interrupted migration; with options.destination, write the converted copy there and leave the source unchanged, reporting through options.onProgress. options.signal cancels it. | | ContainerArchive.estimateMigration(path, options) | Report what migrate() with options.destination would copy and whether it fits, changing nothing. | | IContainerArchiveInputStream | Pipe-compatible ingest/import source contract; requires on(), off(), pipe(), and destroy(), with optional unpipe(). | | ingest(stream, options?) | Store one stream as a snapshot, optionally pinned with options.pin, guarded by options.reserveFreeBytes and cancelled through options.signal. | | ingestMulti(items, options?) | Store several streams as one snapshot, with the same options as ingest(). | | restore(snapshotId, options?) | Return a readable stream for a full snapshot or item; options.signal cancels it. | | listSnapshots(filter?) | List snapshot summaries with their pins, optionally filtered by tags or creation time. | | getSnapshot(id) | Load one full snapshot manifest, items and chunk references included, with its pin. | | deleteSnapshot(id) | Idempotently remove one unpinned restore point while retaining shared immutable pack data. | | pinSnapshot(id, reason) | Durably pin a snapshot so retention and deletion never remove it. | | unpinSnapshot(id) | Idempotently remove a snapshot's pin. | | exportClosure(snapshotIds, options?) | Stream an exact portable closure and return its integrity receipt; options.signal cancels it. | | ContainerArchive.importClosure(path, stream, options) | Verify a closure, construct a new private staging repository, and return the authenticated and validated import receipt; options.reserveFreeBytes guards the destination filesystem. | | ContainerArchiveError | Rejection for every failure the Rust process reports, with a stable code; see Errors. | | ContainerArchiveReserveReachedError | Rejection of an ingest or closure import whose next write would leave less than reserveFreeBytes free. | | ContainerArchiveCancelledError | Rejection of a command cancelled through its signal or a request deadline. | | ContainerArchiveMigrationRequiredError | Rejection of an operation on a repository that needs migrate(). | | verify(options?) | Verify repository integrity; options.signal cancels it. | | prune(retention, dryRun?, options?) | Remove the snapshots retention does not keep and delete whole unreferenced packs; dryRun: true only reports. prune({}, false) only collects garbage. options.signal cancels it. | | repair() | Currently rejects before mutation pending a verified index-generation rebuild. | | reindex() | Currently rejects before mutation pending a verified index-generation rebuild. | | unlock(options?) | Refuses to break a live operating-system lock. | | on(event, handler) | Subscribe to ingest/verify events. | | getCommandQueue() | Report the active command and the commands waiting behind it. | | close() | Reject waiting commands, let the active command finish, close the repository and terminate the Rust process. | | IEngineSelectionOptions | binaryPath for init, open, inspect, migrate, estimateMigration and importClosure: one exact engine executable, failing closed. |

Development

pnpm run build
pnpm test

Useful source entry points:

• ts/index.ts exports the public API.
• ts/classes.containerarchive.ts owns the TypeScript facade and stream socket handling.
• ts/interfaces.ts defines snapshot, retention, verification, repair, and IPC shapes.
• rust/src/main.rs starts the Rust management loop.
• rust/src/ingest.rs, restore.rs, verify.rs, prune.rs, pin.rs, and repair.rs implement the core workflows.

pnpm build compiles the musl target of the host's architecture with @git.zone/tsrust (--locked, local paths remapped). pnpm test and the release build pnpm run build:release compile both musl targets, because the package test checks that the packed package ships both executable; build:release then runs tsrust verify, which fails unless dist_rust/ holds exactly both binaries with provenance of the clean HEAD commit. The builds need the musl C cross compilers x86_64-linux-musl-gcc and aarch64-linux-musl-gcc on PATH for zstd and mimalloc; aarch64-linux-musl-gcc also links the arm64 binary. The Rust binary uses mimalloc as its global allocator because musl's allocator maps and unmaps every chunk-sized buffer, and flate2's zlib-rs backend because musl's memcpy slows the default miniz_oxide inflate; key derivation returns Argon2's working memory to the operating system through mi_collect so it does not stay resident. --cfg=libc_unstable_musl_v1_2_3 selects the libc crate's bindings for the musl 1.2.5 the Rust toolchain ships. Every build verifies native-notices/ first; after a Cargo dependency, toolchain or target change, run pnpm exec tsrust notices, review the diff and commit it (pnpm exec tsrust notices --check verifies offline).

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license.md file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at [email protected].

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.