@bendyline/molen-pack

Content packs for Molen: zip files that carry models, documents and catalogs outside the code packages, opened from wherever your app keeps them.

Part of Molen, an AI-legible 3D experience engine: a deterministic headless kernel, a three.js client, and a dev loop an agent can drive end to end.

Install

npm i @bendyline/molen-pack

Node >= 22.13, ESM only. The . export runs in browsers, Workers and Node; ./node adds file and directory helpers for Node, and ./cache a persistent byte cache with no dependencies at all. The package's only dependencies are @bendyline/molen-schema and fflate.

Use

Where a pack lives is the app's choice: a URL on any origin, a file bundled with an Electron app, a file the user picked, bytes you already hold, or storage you read yourself.

import { createPackSet, openPack } from '@bendyline/molen-pack';

const packs = createPackSet([
  await openPack('/packs/molen.entities-3f2a9c1e5b7d.zip'), // any URL
  await openPack(userPickedFile), // a File or Blob
  await openPack(bytes), // an ArrayBuffer or Uint8Array
  await openPack({ size, read: (offset, length) => myStorage.read(offset, length) }),
]);

// Asset ids, pack-qualified paths, or plain paths; the pack added last wins.
const glb = await packs.readBytes('molen.entities.aircraft.p51d');
const types = await packs.readJson('pack:molen.entities/types/aircraft.types.json');

// The client's asset provider shape, so models and material docs load from packs.
const client = await createClient(link, { assets: { provider: packs.assetProvider() } });

Opening a remote pack reads only its last 64 KiB, which holds the manifest, the zip directory and the small documents. Other files are read on demand with HTTP range requests; reads issued together are merged when they sit close in the file, at most six run at once, and transient failures are retried. A server that ignores Range just sends the whole file, which is used as is.

Molen's own packs (entities, the default style pack, the Earth catalogs, the star table) are published at molen.dev/packs. npx molen pack fetch https://molen.dev/packs/index.json downloads them and pins each one's contentHash in project.json.

Build your own packs with the CLI (molen pack build <dir> --out-dir <d>) or in code:

import { buildPack, openDirPack } from '@bendyline/molen-pack/node';

// A directory with a molen-pack.source.json describing the pack.
const built = await buildPack('content/my-props', { outDir: 'dist/packs' });
built.file; // 'my.props-3f2a9c1e5b7d.zip', named by a hash of its bytes

// The same content, unzipped: handy in tests and during development.
const pack = await openDirPack('content/my-props');

Host packs unzipped

A CDN serves a pack just as well with its files loose: every file is its own cacheable URL, and a new release uploads only the packs that changed. hostPacks publishes the packs a pack index lists that way. Each version goes to <pack id>/<12 hex of its contentHash>/ once and never changes, so those files can be served as immutable, and the index.json it writes last is the only document that changes between releases:

import { hostPacks } from '@bendyline/molen-pack/node';

// dist/packs/index.json lists built zips; _a/ becomes the directory to upload.
await hostPacks('dist/packs/index.json', { outDir: '_a' });
// _a/index.json → { "molen.sky": { "file": "molen.sky/b42226982545/molen-pack.json", … } }

openPack (and openPacksFromIndex in @bendyline/molen-earth) opens a URL ending in molen-pack.json as an unzipped pack, reading each file from its path beside the manifest. writeDirectoryPack(pack, dir) writes a single pack in that layout.

Cache what you read

./cache keeps the bytes of range-read archives (packs, PMTiles) across visits. Each archive is split into 64 KiB blocks held in memory and in IndexedDB; neighbouring misses become one range request, concurrent reads share blocks, and the least recently used blocks go once the store passes its budget.

import {
  cachingRangeReader,
  createBlockCache,
  createIndexedDbByteStore,
  createMemoryByteStore,
  urlRangeReader,
} from '@bendyline/molen-pack/cache';
import { openPack } from '@bendyline/molen-pack';

const cache = createBlockCache(
  createIndexedDbByteStore({ budgetBytes: 300 * 1024 * 1024 }) ?? createMemoryByteStore(),
);
// Key a pack by its content hash so a cached copy can never be another version.
const reader = urlRangeReader(url, { size: entry.size });
const pack = await openPack(cachingRangeReader(reader, cache, { key: entry.contentHash, url }));

cachingRangeSource(source, cache, archive) does the same for a PMTiles Source, and cachingDocumentFetch(cache.store) wraps fetch so small JSON documents are revalidated with If-None-Match instead of refetched and still load offline. A stable URL whose file is replaced in place is cached with keyBy: 'etag': each server version is its own archive and older ones are deleted. Store failures (private browsing, quota) only cost network, never a failed read.

What's in it

• openPack(input, options) — a Pack from a URL, Blob, bytes, or a RangeReader, with readBytes / readText / readJson / prefetch. Options cover whole-or-range mode, retries, concurrency, progress, an expected contentHash, and sha256 checks per file. openDirectoryPack(url) opens an unzipped pack from its molen-pack.json.
• createPackSet(packs) — several packs read as one, with later packs overriding earlier ones, provided(role) to find the files a pack declares for a role, and assetProvider().
• createPack(files, options) / describePack / packFromFiles — build a pack, or a pack-shaped view of loose files, from bytes in any environment.
• ./node: buildPack, openFilePack, openDirPack, openPackAt, readPackSource, extractPack, writeDirectoryPack, hostPacks.
• ./cache: createBlockCache, createIndexedDbByteStore, createMemoryByteStore, cachingRangeSource, cachingRangeReader, urlRangeReader, cachingDocumentFetch.

A pack is a standard zip you can open with any zip tool. Binary files are their own members, deflated when that helps. Small text files are concatenated into a few deflated solid blocks, which compresses JSON far better than one file at a time: the default worldgen style pack, 240 files and 799 KB, builds to 79 KB. molen pack extract turns a pack back into its source files. The manifest (molen/pack@1) lists every file with its size and sha256, the asset ids it provides, and a contentHash over paths and contents that does not depend on the zip layout.

Status

0.x, versioned independently of the other @bendyline/molen-* packages. The format is beta. Packs are written without zip64, so a pack must stay under 4 GiB and 65,535 members. Building the same files always produces the same bytes; that depends on the exact fflate version this package pins.

Docs

• Schema: molen/pack@1
• Schema: molen/pack-source@1

MIT © Bendyline LLC