@xemahq/deterministic-archive

This package belongs to Layer 1 — a pure SDK with zero runtime dependencies (node builtins only; everything else is dev tooling). It depends on no first-party package at all, so any Layer 2 service and any Layer 3 biome may use it without crossing the release DAG.

What it is

A byte-deterministic USTAR + gzip writer. Give it a set of named buffers and it returns the archive; give it the same set again, anywhere, and it returns the same bytes.

Why determinism is the whole contract

Callers content-address the result — the store keys a bundle by the sha256 of these bytes. So anything that varies between two runs over the same inputs turns a re-upload into a spurious "changed" artifact, and it does so silently: the archive is still valid, still unpacks, still passes every test that checks its contents. The breakage surfaces months later as a diff nobody can explain.

A general-purpose tar library fills exactly the fields that must not vary:

| Field | A normal writer takes it from | Here | |---|---|---| | mtime | the wall clock | pinned to 0 | | uid / gid | the process | pinned to 0 | | mode | the filesystem umask | pinned to 0o644 | | entry order | directory iteration / settle order | normalised by sort | | gzip MTIME | the wall clock | zero (Node's gzipSync leaves it) |

Each of those is asserted in this package's tests rather than assumed — the uid/gid assertion runs a control first, confirming the test process is not root, so it cannot pass by coincidence.

Usage

import { writeDeterministicTarGz } from '@xemahq/deterministic-archive';

const archive = writeDeterministicTarGz({
  gzipLevel: 9,
  entries: [
    { name: 'manifest.json', body: manifestBytes },
    { name: 'config.yaml', body: configBytes },
  ],
});

writeDeterministicTar returns the uncompressed archive if you want to gzip it yourself.

gzipLevel is required on purpose

The level changes the output bytes, and callers content-address those bytes. A default would let a package upgrade silently re-key every artifact somebody had already stored, so each call site states its level and keeps it.

sortKey — when the canonical order is not an order over the name

Entries are sorted before they are written, so the same set produces the same bytes regardless of the order the caller assembled the array in. By default the sort key is the entry name.

That is wrong for a caller whose names are positional — members/0, members/1, … — because such names are assigned after sorting on some other key, and members/10 sorts before members/2. Those callers state the key they actually canonicalised on:

entries: ordered.map((entry, position) => ({
  ...entry,
  sortKey: String(position).padStart(6, '0'),
})),

Sort keys must be unique: two entries sharing one have no defined order between them, which is the nondeterminism this package exists to remove, so it is refused rather than resolved by input order.

What it refuses

Fail-fast, with no "encode it anyway" branch — a refused input is one where the archive would be unsafe to unpack or would not be the archive that was asked for:

• an entry name that is absolute, contains .., ., an empty segment, a backslash or a NUL (validated at write time, so a bad name is the producer's failing test rather than a path traversal in whoever unpacks it);
• a name longer than the 99 UTF-8 bytes the ustar name field holds — this writer deliberately emits no GNU long-name or PAX extension record;
• duplicate entry names, and duplicate sort keys;
• a body too large for the 12-byte octal size field, which would otherwise emit a header that parses as a different size;
• a gzipLevel outside 0–9.

Scope

Writing only. There is no reader here: a reader's interesting decisions are its limits (entry caps, byte caps, which type flags to accept), and those belong to the consumer that knows what it is unpacking.