jampak

v2.0.0

Published

2 months ago

An efficient file storage solution specifically made for both JavaScript and TypeScript data types in Node.js with a focus on accuracy, expandability, security, and performance.

0High
0Medium
0Low

hearhellacopters

buffer parser reader json decode unpack struct

JamPak for Node/JavaScript/TypeScript

JamPak is an efficient file storage solution specifically made for both JavaScript and TypeScript data types in Node.js with a focus on accuracy, expandability, security, and performance. Includes compact storage of all JSON types as well as TypedArrays, Maps, Sets, Dates, Symbols and more! This library uses a heavily modified implementation of MessagePack on top of its own binary reader/writer to improve storage size and create efficient binary serialization.

JamPak improvements over MessagePack:

Reduced file size by splitting data into two sections, values and strings
Keys can be stripped for 'schema' like control to further reduce file size and add security
Compression / authenticated encryption / CRC check
Endianness control
Full Symbol support, including registered and well-known symbols

Zero runtime dependencies. Only Node's own fs, zlib, crypto and buffer are used.

*Note: Only compatible with Node.js.

Upgrading from 1.x? Version 2 changes the file format, the encryption scheme and the extension API. See Migrating from 1.x.

How it works

JamPak's secret is it uses JavaScript's own Set feature to create a unique array of all string data in the file to cut down on size. Saving index numbers in place of the strings for repeated use. Great for data with large repeating object arrays or "table like" data where string keys are repeated. The file splits the data into two sections, the value section and the string section. The file can be futher compressed with zlib and even encrpyted as well as an optional CRC32 check.

Here is a breakdown of a sample file struture.

Synopsis

import { JPEncode, JPDecode } from "jampak";

const object = {
  null: null,
  undefined: undefined,
  integer: 1,
  float: Math.PI,
  bigint: 0x100000000000000n,
  string: "Hello, world!",
  array: [10, 20, 30],
  object: { foo: "bar" },
  mapExt: new Map([["key1","data1"],["key2","data2"]]),
  setExt: new Set([50, 60, 70]),
  symbolExt: Symbol("symbol"),
  regexExt: /(regex)/g,
  bufferExt: Buffer.from([1, 2, 3]),
  uint8arrayExt: new Uint8Array([1, 2, 3]),
  dateExt: new Date()
};

const encoder = new JPEncode();

const encoded: Buffer = encoder.encode(object);

// Decoding without options gives every type back as it went in:
// Maps stay Maps, Dates stay Dates, Symbol.for round-trips to the same symbol.
const exact = new JPDecode().decode(encoded);

// Or flatten everything into plain JSON:
const decoder = new JPDecode({ makeJSON: true });

const plain = decoder.decode(encoded);

console.log(plain);
// {
//   null: null,
//   undefined: 'undefined',
//   integer: 1,
//   float: 3.141592653589793,
//   bigint: '72057594037927936',
//   string: 'Hello, world!',
//   array: [ 10, 20, 30 ],
//   object: { foo: 'bar' },
//   mapExt: [ [ 'key1', 'data1' ], [ 'key2', 'data2' ] ],
//   setExt: [ 50, 60, 70 ],
//   symbolExt: { symbolGlobal: false, symbolKey: 'symbol', symbolKind: 'Unique' },
//   regexExt: { regexSrc: '(regex)', regexFlags: 'g' },
//   bufferExt: { type: 'Buffer', data: [ 1, 2, 3 ] },
//   uint8arrayExt: { '0': 1, '1': 2, '2': 3 },
//   dateExt: '2025-07-10T02:17:53.721Z'
// }

Install

This library is published to npmjs.com as jampak.

npm install jampak

API

`new JPEncode(EncoderOptions?)`

Creates a new JPEncode class with set EncoderOptions. You can then encode your data with encoder.encode(data) into a single JamPak-encoded Buffer. It throws errors if data is, or includes, a non-serializable object such as a function or other types not added to the extensions.

Alternatively you can include a filePath string as a second argument and write the file out directly.

For example:

import { JPEncode } from "jampak";

const encoder = new JPEncode({ encrypt: true, stripEncryptKey: true });

const data = { foo: "bar" };

encoder.encode(data, "./foo.jpk"); // Saves the encrypted Buffer directly to file

// Nothing that can open the file was stored, so keep this or lose the data.
const key: Buffer = encoder.encryptionKey;

`new JPEncodeAsync(EncoderOptions?)`

Creates a new JPEncodeAsync class with set EncoderOptions. You can then encode your data with await encoder.encode(data) into a single JamPak-encoded Buffer. It throws errors if data is, or includes, a non-serializable object such as a function or other types not added to the extensions.

Alternatively you can include a filePath string as a second argument and write the file out directly.

For example:

import { JPEncodeAsync } from "jampak";

const encoder = new JPEncodeAsync({ encrypt: true, stripEncryptKey: true });

const data = { foo: "bar" };

await encoder.encode(data, "./foo.jpk"); // Saves the encrypted Buffer directly to file

// Nothing that can open the file was stored, so keep this or lose the data.
const key: Buffer = encoder.encryptionKey;

`EncoderOptions`

| Name | Type | Default | Desc | | ------------------- | ---------------- | ----------------------------- | --- | | extensionCodec | JPExtensionCodec | ExtensionCodec.defaultCodec | User added extension types, see Extension Types. | | endian | string | "little" | Change the endianness of the Buffer writing. | | encrypt | boolean | false | If the file should be encrypted (AES-256-GCM). | | password | string | Buffer | undefined | Passphrase to derive the encryption key from, via scrypt. Pair with stripEncryptKey if the point is that only you can open the file. | | key | Buffer | undefined | A 32-byte AES key used as-is, skipping scrypt. Use when you already hold high-entropy key material — scrypt costs a fixed ~50 ms per encode and per decode. Generate one with crypto.randomBytes(32). | | stripEncryptKey | boolean | false | Withhold the AES key from the file. This is the only switch between the two modes — see How the AES key works. Off: the file opens itself. On: you must keep the password or the encryptionKey the class reports. | | CRC32 | boolean | false | Add a CRC32 check to the file. | | compress | boolean | false | Compress the file's data. | | stripKeys | boolean | false | Remove all keys from the save file. Must save the keysArray from the class it was created from. | | initialSize | number | 4096 | Bytes allocated up front per write buffer. | | growthIncrement | number | 0x1000000 | Maximum bytes added in one growth step. Growth is otherwise geometric. | | includeSymbolKeys | boolean | true | Encode symbol-keyed enumerable properties of plain objects. | | maxDepth | number | 512 | Nesting depth at which encoding aborts, guarding against stack exhaustion. | | msgpack | boolean | false | Store the payload as real MessagePack instead of the JamPak layout. NOTE: Does not support symbols! |

Class `JPEncode` functions

Note: Outside of the basic encode, these functions should only be used within a user created Extension Type.

| Functions | Type | Desc | | ------------------------- | -------------------------------------------------- | --- | | encode(object, filePath?) | function (uknown, string?) : Buffer | The basic function that creates the JamPak Buffer. If a filePath is supplied, it writes the file directly out. | | encodeObject(valueWriter, object, depth?) | function (JPWriter, Record<string, unknown>, number?): number | Encodes a Object to the passed JPWriter's buffer. Returns the number of bytes written. | | encodeArray(valueWriter, array, depth?) | function (JPWriter, Array<unknown>, number?): number | Extension function use only. Encodes a Array to the passed JPWriter's buffer. Returns the number of bytes written.| | encodeString(valueWriter, string, isKey?) | function (JPWriter, string, boolean?): number | Extension function use only. Encodes a string to the string section of the current file and writes the index to the passed JPWriter's buffer. Returns the number of bytes written to the buffer. | | encodeNull(valueWriter) | function (JPWriter): number | Extension function use only. Encodes a null to the passed JPWriter's buffer. Returns the number of bytes written. | | encodeUndefined(valueWriter) | function (JPWriter): number | Extension function use only. Encodes a undefined to the passed JPWriter's buffer. Returns the number of bytes written.| | encodeBoolean(valueWriter) | function (JPWriter): number | Extension function use only. Encodes a true or false to the passed JPWriter's buffer. Returns the number of bytes written.| | encodeFinished(valueWriter) | function (JPWriter): number | Extension function use only. Encodes a "finished" byte to the passed JPWriter's buffer. Will end all looping when the reader hits this byte. Returns the number of bytes written. | | encodeListEnd(valueWriter) | function (JPWriter): number | Extension function use only. Encodes a "list end" byte to the passed JPWriter's buffer, useful when pulling loose data and don't want to break the whole loop. Returns the number of bytes written. | | encodeNumber(valueWriter, number) | function (JPWriter, number): number | Extension function use only. Encodes a number to the passed JPWriter's buffer. Computes the right byte size base on value. Returns the number of bytes written.| | encodeBigInt64(valueWriter, bigint) | function (JPWriter, bigint): number | Extension function use only. Encodes a bigint to the passed JPWriter's buffer. Always written as a 64 bit value.|

Class `JPEncode` objects

After encode has run.

| Name | Type | Desc | ------------------------- | ------------------------------------------ | -------------------------------------------------- | | encryptionKey | Buffer | null | The 32-byte AES key this file uses — supplied, derived from your password, or generated. Always set after an encrypted encode. Must be saved if stripEncryptKey was used (or keep the password instead). | | password | string | Buffer | null | The passphrase you supplied, or null. Never written to the file. | | keysArray | string[] | The keys for the object data. Must be saved if stripKeys was used. | | CRC32Hash | number | The computed CRC32 hash if enabled in options | | errored | boolean | Whether any non-fatal problem was recorded. | | errorMessage | string | Those problems, newline separated. |

`new JPDecode(DecoderOptions?)`

Creates a new JPDecode class with set DecoderOptions. You can then decode your data with decoder.decode(data) from a single JamPak-encoded Buffer and returns the decoded object typed unknown. If the type of data passed to decode is a string it will assume it is a file path and try to read the file data directly.

For example:

import { JPDecode } from "jampak";

const decoder = new JPDecode({ key }); // or { password }, if you used one

const object = decoder.decode('./foo.jpk');

console.log(object);

`new JPDecodeAsync(DecoderOptions?)`

Creates a new JPDecodeAsync class with set DecoderOptions. You can then decode your data with await decoder.decode(data) from a single JamPak-encoded Buffer and returns the decoded object typed unknown. If the type of data passed to decode is a string it will assume it is a file path and try to read the file data directly.

For example:

import { JPDecodeAsync } from "jampak";

const decoder = new JPDecodeAsync({ key }); // or { password }, if you used one

const object = await decoder.decode('./foo.jpk');

console.log(object);

`DecoderOptions`

| Name | Type | Default | Desc | | --------------- | ------------------- | ----------------------------- | ---- | | extensionCodec | JPExtensionCodec | ExtensionCodec.defaultCodec | User added extension types, see Extension Types. | | keysArray | string[] | [] | String array from when stripKeys was used during encoding. | | password | string | Buffer | undefined | Passphrase used at encode time. Only needed for files written with stripEncryptKey. | | key | Buffer | undefined | The 32-byte AES key used at encode time. Only needed for files written with stripEncryptKey. |

Types to JSON table

Type conversion when using makeJSON in the decoder.

| Type | Conversion | | --------- | -------------------------------------------------- | |undefined | "undefined" string | |RegExp | {regexSrc: string, regexFlags: string} object | |symbol | {symbolGlobal: boolean, symbolKey: string, symbolKind: string} object* | |bigint | number if safe, otherwise string | |Set | Array | |Map | Array[] | |Date | ISO 8601 string | |Buffer | {type: "Buffer", data: number[]} object | |TypedArray | object keyed by index | | symbol keys | dropped, as JSON.stringify does |

symbolKind is one of "Registered", "WellKnown", "Unique" or "UniqueAnonymous".

NOTE: symbol will error in msgpack mode.

Note: If you create Extension Types, you must handle the conversion in your decode function.

Class `JPDecode` functions

Note: Outside of the basic decode, these functions should only be used within a user created Extension Type. Using the JPDecodeAsync class returns a promise instead.

| Functions | Type | Desc | | ------------------------- | -------------------------------------------------- | --- | | decode(bufferOrSourcePath) | function (Buffer \| string) : unknown | Your Buffer to decode or the source path to a JamPak file. | The function that decodes the JamPak Buffer. | | doDecode(bufferOrReader) | function (Buffer \| JPReader): unknown | Extension function use only. Runs a raw decode on the passed JPReader's buffer. Return data wherever it ends based on the start value. |

Class `JPDecode` objects

After decode or decodeAsync as run.

| Name | Type | Desc
| ------------------------- | ------------------------------------------ | -------------------------------------------------- | symbolList | symbol[] | Any symbol created on decode are in this array.* | | hasExtensions | boolean | If the returned data had any extension types used. | | validJSON | boolean | If the decoded data is already valid JSON | | CRC32OnFile | number | The CRC32 hash on file. | | CRC32Hash | number | The computed CRC32 hash of the file. | | errored | boolean | Whether any non-fatal problem was recorded (CRC mismatch, an unresolvable string index, a size that did not match the header). | | errorMessage | string | Those problems, newline separated. Nothing is written to the console. |

NOTE: symbol will error in msgpack mode.

`inspectHeader(buffer)`

Reads a file's header without decoding it. Use it to check a file is readable before committing to a decode — in particular to tell a version mismatch apart from corruption, and to find out whether a password or keysArray will be needed.

Only the header is examined, so the first 84 bytes of a large file are enough.

import { inspectHeader } from "jampak";

const info = inspectHeader(fs.readFileSync("./data.jpk"));

if (!info.supported) {
  throw new Error(info.reason); // e.g. written by format 1.x, or by a newer major
}

if (info.needsSecret) { /* prompt for a password */ }

if (info.needsKeysArray) { /* load the saved keysArray */ }

| Field | Type | Desc | | ----- | ---- | ---- | | endian | "little" \| "big" | Byte order the file was written in. | | versionMajor / versionMinor | number | Format version stored in the file. | | supported | boolean | Whether this package can decode it. | | reason | string? | Why not, when supported is false. | | needsSecret | boolean | Encrypted with stripEncryptKey, so a key or password must be supplied. | | needsKeysArray | boolean | Written with stripKeys — the saved keysArray must be supplied. | | headerSize | number | Total header bytes, including the optional trailer. | | flags | JPFlags | The file's flag bits. | | valueSize / strSize / dataSize | bigint | Section sizes from the header. |

Throws only when the input is not a JamPak file at all (bad magic, too short). A version mismatch is reported through supported, not thrown.

Format versions

The header carries a major and minor format version, and the decoder checks them before reading anything whose meaning depends on the layout:

| File version | Behaviour | | ------------ | --------- | | Older major (e.g. 1.x) | Throws JPVersionError — the layout differs, so there is nothing useful to attempt. | | Newer major | Throws JPVersionError — upgrade the package. | | Newer minor | Decodes, with a warning on errorMessage. Minor bumps only add things that do not move existing bytes; an unknown extension type decodes to JPExtData. | | Same or older minor | Decodes silently. |

This matters because the magic bytes did not change between 1.x and 2.x. Without the check a 1.x file parses as a malformed 2.x file and fails somewhere deep in the value section with a misleading "corrupt file" message.

JPVersionError carries fileMajor, fileMinor, packageMajor, packageMinor and direction ("older" or "newer"), so you can tell "go get a different package version" apart from a genuinely damaged file:

import { JPDecode, JPVersionError } from "jampak";

try {
  new JPDecode().decode(buffer);
} catch (err) {
  if (err instanceof JPVersionError) {
    console.error(`Need jampak for format ${err.fileMajor}.x, have ${err.packageMajor}.${err.packageMinor}`);
  } else {
    throw err;
  }
}

Exported alongside it: VERSION_MAJOR, VERSION_MINOR and MIN_SUPPORTED_VERSION_MAJOR.

Extension Types

To handle Extension Types, this library provides JPExtensionCodec class.

This is an example to setup custom extension types that handles Date classes in TypeScript:

import { JPDecode, JPEncode, JPExtensionCodec, JPExtensionType, JPReader, JPWriter } from "jampak";

// Note this is an example, `Date` handling is built in.

/**
 * Example number type to register the extension between 0x00 - 0xCF.
 *
 * 0xD0 - 0xFF are reserved for internal use, so a value in that range is
 * rejected by `register`.
 */
const DATE_EXT_TYPE = 0xC0;

/**
 * Example encoding function
 *
 * @param input - Your object to type check and encode
 * @param encoder - class encoder
 * @param context - Context of the class
 * @returns The extension payload, or `null` if `input` is not your type.
 */
function encodeTimestampExtension<ContextType = undefined>(
  input: unknown,
  encoder: JPEncode<ContextType>,
  context: ContextType): Buffer | null {
  // check if the input is the same type, else return null
  if (!(input instanceof Date)) {
    return null;
  }

  const TIMESTAMP32_MAX_SEC = 0x100000000 - 1; // 32-bit unsigned int

  const TIMESTAMP64_MAX_SEC = 0x400000000 - 1; // 34-bit unsigned int

  const msec = input.getTime();

  const _sec = Math.floor(msec / 1e3);

  const _nsec = (msec - _sec * 1e3) * 1e6;
  // Normalizes { sec, nsec } to ensure nsec is unsigned.
  const nsecInSec = Math.floor(_nsec / 1e9);

  const sec = _sec + nsecInSec;

  const nsec = _nsec - nsecInSec * 1e9;
  // Build the payload with a JPWriter, in the encoder's endianness.
  const bw = new JPWriter({ initialSize: 12, little: encoder.endian !== "big" });

  if (sec >= 0 && nsec >= 0 && sec <= TIMESTAMP64_MAX_SEC) {
    if (nsec === 0 && sec <= TIMESTAMP32_MAX_SEC) {
      // timestamp 32 = { sec32 (unsigned) }
      bw.u32(sec);

      return bw.toBuffer();
    }

    // timestamp 64 = { nsec30 (unsigned), sec34 (unsigned) }
    const secHigh = Math.floor(sec / 0x100000000);

    const secLow = sec >>> 0;

    bw.u32(((nsec << 2) | (secHigh & 0x3)) >>> 0);

    bw.u32(secLow);

    return bw.toBuffer();
  }

  // timestamp 96 = { nsec32 (unsigned), sec64 (signed) }
  bw.u32(nsec >>> 0);

  bw.i64(BigInt(sec));

  return bw.toBuffer();
};

/**
 * Example decoding function
 *
 * @param data - `JPReader` over the extension payload.
 * @param decoder - class decoder
 * @param extensionType - Registered extension number between 0x00 - 0xCF (for dummy checks)
 * @param context - Context of the class (shouldn't be needed)
 */
function decodeTimestampExtension<ContextType = undefined>(
  data: JPReader,
  decoder: JPDecode<ContextType>,
  extensionType: number,
  context: ContextType): Date {
    // check if the type matches
    if (extensionType != DATE_EXT_TYPE) {
      throw new Error(`Extension for Date encoding 0x${extensionType.toString(16).padStart(2, "0")} does not match register type 0x${DATE_EXT_TYPE.toString(16).padStart(2, "0")}`);
    }
    // data may be 32, 64, or 96 bits
    switch (data.size) {
      case 4: {
        // timestamp 32 = { sec32 }
        const sec = data.u32();

        return new Date(sec * 1e3);
      }
      case 8: {
        // timestamp 64 = { nsec30, sec34 }
        const nsec30AndSecHigh2 = data.u32();

        const secLow32 = data.u32();

        const sec = (nsec30AndSecHigh2 & 0x3) * 0x100000000 + secLow32;

        const nsec = nsec30AndSecHigh2 >>> 2;

        return new Date(sec * 1e3 + nsec / 1e6);
      }
      case 12: {
        // timestamp 96 = { nsec32 (unsigned), sec64 (signed) }
        const nsec = data.u32();

        const sec = Number(data.i64());

        return new Date(sec * 1e3 + nsec / 1e6);
      }
      default:
        throw new Error(`Unrecognized data size for timestamp (expected 4, 8, or 12): ${data.size}`);
    }
};

/**
 * Example object passed to `JPExtensionCodec.register`
 */
export const timestampExtension: JPExtensionType = {
  type: DATE_EXT_TYPE,
  encode: encodeTimestampExtension,
  decode: decodeTimestampExtension,
  // dummy functions for async, not in use here
  encodeAsync: async () => null,
  decodeAsync: async () => undefined,
};

const ExtCodec = new JPExtensionCodec();

ExtCodec.register(timestampExtension);

const encoder = new JPEncode({ extensionCodec: ExtCodec });

const encoded = encoder.encode(new Date());

const decoder = new JPDecode({ extensionCodec: ExtCodec });

const decoded = decoder.decode(encoded);

Ensure you include your extensionCodec in any recursive encode and decode statements!

Note that extension types for custom objects must be 0x00 - 0xCF, while 0xD0 - 0xFF is reserved for JamPak itself.

ExtensionCodec context

When you use an extension codec, it might be necessary to have encoding/decoding state to keep track of which objects got encoded/re-created. To do this, pass a context to the EncoderOptions and DecoderOptions:

import { JPDecode, JPEncode, JPExtensionCodec, JPWriter } from "jampak";

class MyContext {
  track(object: any) { /*...*/ }
}

class MyType { /* ... */ }

const ExtCodec = new JPExtensionCodec<MyContext>();

// MyType
const MYTYPE_EXT_TYPE = 0 // Any in 0x00 - 0xCF
ExtCodec.register({
  type: MYTYPE_EXT_TYPE,
  encode: (object, encoder, context) => {
    if (object instanceof MyType) {
      context.track(object);

      // Build the payload with your own writer, then hand back its bytes.
      const bw = new JPWriter({ little: encoder.endian !== "big" });

      encoder.encodeObject(bw, object.toJSON());

      return bw.toBuffer();
    } else {
      return null;
    }
  },
  decode: (data, decoder, extType, context) => {
    const decoded = decoder.doDecode(data);
    const my = new MyType(decoded);
    context.track(my);
    return my;
  },
  encodeAsync: async () => null,
  decodeAsync: async () => undefined,
});

// and later
const context = new MyContext();

const encoder = new JPEncode({ extensionCodec: ExtCodec, context: context });

const encoded = encoder.encode({ myType: new MyType<any>() });

const decoder = new JPDecode({ extensionCodec: ExtCodec, context: context });

const decoded = decoder.decode(encoded);

Advanced Features

JamPak has four major features: encryption, compression, key stripping and CRC check.

encrypt
- Everything after the header is encrypted with AES-256-GCM. The key is derived with scrypt from a per-file random 128-bit salt; the 96-bit nonce is also random per file. The salt, nonce and 128-bit authentication tag are stored in the header — all three are public values that GCM needs in order to decrypt, and none of them reveal the key.
- Because GCM is authenticated, a modified file fails to decrypt rather than decoding into plausible garbage. You do not need CRC32 to detect tampering; use it to catch accidental corruption of an unencrypted file.
- There is exactly one decision to make, stripEncryptKey: either the file opens itself, or only you can open it. Everything else is a detail of where the key came from. See How the AES key works.
- Cost: with a key, encryption adds roughly 5% to encode time (AES-GCM runs at ~1 GB/s). With password, add ~50 ms per call for scrypt — deliberate, and a fixed cost regardless of document size.
compress
- Outside of the 32 byte header, the file is compressed in 512kb zlib chunks. You can also encrypt the compressed file as well (encryption happens after compression)
- The amount of data saved depends on the size of the file and the type of data within.
stripKeys
- More for security than size savings, this creates a schema like file where the keys to the data aren't include in the file. The keys can be found in the keysArray object in the class after encoding and MUST be saved or the file won't be able to be decrypted.
CRC32
- Outside of the header, runs a CRC32 hash over the values and string data — before compression and encryption, so it validates the data you actually get back. Saves the hash to the file. A mismatch is reported through errored / errorMessage rather than thrown, so a partly-readable file can still be inspected.
- CRC32 is an error-detecting code, not a MAC. It catches accidental corruption; it does not detect deliberate tampering. Use encrypt for that.

How the AES key works

Everything after the header is encrypted with AES-256-GCM under a 32-byte key. There is one decision to make, and it is the one you would expect: do you want to be the only person who can open this file, or should the program handle it for you?

                    where the key comes from
  ┌───────────────────────────────────────────────────────┐
  │  key: Buffer(32)   ──────────────────────▶ used as-is │
  │  password: string  ── scrypt(pw, salt) ─▶ derived     │
  │  neither           ── randomBytes(32) ──▶ generated   │
  └───────────────────────────┬───────────────────────────┘
                              │  32-byte AES key
                              ▼
            ┌─────────────────────────────────┐
            │  stripEncryptKey?               │
            ├────────────────┬────────────────┤
            │  false         │  true          │
            │  (default)     │                │
            ├────────────────┼────────────────┤
            │ key IS written │ key is NOT     │
            │ to the file    │ written        │
            │                │                │
            │ opens with no  │ opens only     │
            │ input at all   │ with your      │
            │                │ key/password   │
            └────────────────┴────────────────┘

Whichever route the key took leaves no trace in the file. A file records only whether the key is stored, not whether it was typed, generated or derived.

The two modes

| | stripEncryptKey: false (default) | stripEncryptKey: true | | --- | --- | --- | | Intent | "hide this from casual view" | "only I can open this" | | Key in file | yes | no | | To decode | nothing needed | the key or password | | Security | obfuscation — the key ships with the data | real, as strong as your secret |

import crypto from "node:crypto";
import { JPEncode, JPDecode } from "jampak";

// "Let the program handle it." A 256-bit key is generated and stored.
const encoder = new JPEncode({ encrypt: true });

encoder.encode(data, "./hidden.jpk");

new JPDecode().decode("./hidden.jpk"); // just works

// "Only I can open this."
const key = crypto.randomBytes(32);

new JPEncode({ encrypt: true, key, stripEncryptKey: true }).encode(data, "./secret.jpk");

new JPDecode({ key }).decode("./secret.jpk");

// Same, with a passphrase instead of key material.
const enc = new JPEncode({ encrypt: true, password: "correct horse battery staple", stripEncryptKey: true });

enc.encode(data, "./secret2.jpk");

new JPDecode({ password: "correct horse battery staple" }).decode("./secret2.jpk");

After encoding, the class hands back what it used, whether you supplied it or it was generated:

encoder.encryptionKey; // Buffer(32) — the actual AES key, always set when encrypted
encoder.password;      // what you passed, or null

Save encryptionKey (or your password) whenever you used stripEncryptKey, or the data is unrecoverable. Supply either one to the decoder; both open the file.

Passing both key and password throws. Only one could be used and the other would be silently ignored — which is how you end up saving the wrong value.

If you supply a secret but don't set stripEncryptKey, the encode succeeds and notes on errorMessage that the file will open without that secret. That is the documented default, but it is rarely what someone who bothered to pick a password intended.

What is on disk

When Encrypted is set the header always carries a fixed 44-byte block, then the key only if it was not stripped:

| Bytes | Field | Present | | ----- | ----- | ------- | | 16 | salt | always — scrypt input, random per file | | 12 | iv | always — GCM nonce, random per file | | 16 | tag | always — GCM authentication tag | | 32 | key | only when stripEncryptKey is off |

HEADER_SIZE is therefore 108 for a self-opening encrypted file and 76 for a stripped one (add 4 to either when CRC32 is on).

None of the salt, IV or tag is secret. GCM needs all three to decrypt and verify, and none reveals anything about the key — that is why they sit in the clear. They are written as one unconditional block so a parser never has to work out which parts are present; the 32-byte key is the only conditional field, and it maps exactly to the one decision above.

Why the salt and IV are random per file. The salt means one passphrase produces a different key in every file, so a single precomputed table cannot attack a corpus. The random IV means encrypting the same data twice produces different ciphertext. Version 1 derived both from the key, so identical inputs produced byte-identical files and reused the nonce — the one thing GCM must never do.

Why GCM rather than CBC. The tag authenticates the ciphertext, so a modified file fails to decrypt with a clear error instead of decoding into plausible-looking garbage. You do not need CRC32 to detect tampering on an encrypted file.

What is not protected. The base 32-byte header is in the clear, so the flags, version and section sizes are visible to anyone holding the file. Only the payload is encrypted.

Limits and hardening

Decoding treats its input as untrusted:

| Guard | Default | Option | | ----- | ------- | ------ | | Container element count | 16,777,216 | maxContainerSize | | Nesting depth | 512 | maxDepth | | Declared sizes | Validated against bytes remaining | — | | __proto__ as an object key | Rejected | — | | Circular input (encode) | Rejected | — |

Sizes are checked against the bytes actually present before anything is allocated, so a twelve-byte file claiming a four-billion-element array is rejected rather than acted on.

JamPak Specification

This library is based around the MessagePack specification (head byte, optional size, then data), but modified and expanded to better fit JavaScript and TypeScript data types. It mindfully encodes data first by JSON standard types (object, array, number, boolean, string, null) then extends into other types:

[x] Keys separation, for key stripping for extra security.
[x] Kill byte, so the decoder knowns when the data is finished.
[x] bigint always encodes to 64 bit but will return as a number if within safe number range.
[x] Map ext type (NOT object)
[x] Set ext type
[x] Symbol ext type, including registered, well-known and unique symbols — see Symbol support
[x] TypedArray ext type (from BigUint64Array to Uint8ClampedArray)
[x] Buffer ext type
[x] Date ext type
[x] RegExp ext type

NOTE: Symbol will error in msgpack mode.

Symbol support

JavaScript has three kinds of symbol, and they need different treatment to survive a round trip:

| Kind | Example | Stored as | Restores to | | ---- | ------- | --------- | ----------- | | Registered | Symbol.for("id") | The registry key | The identical symbol — Symbol.for is global | | Well-known | Symbol.iterator | The property name on Symbol | The identical symbol — it is a realm intrinsic | | Unique | Symbol("desc") | A per-document id plus the description | A new symbol, but the same one for every occurrence | | Unique, anonymous | Symbol() | A per-document id | A new symbol, same rule as above |

The per-document id is what makes reference identity hold:

const shared = Symbol("shared");

const decoded = new JPDecode().decode(new JPEncode().encode({ a: shared, b: shared }));

decoded.a === decoded.b; // true

Descriptions are interned in the file's string section like any other string, so a symbol repeated across a large document costs a few bytes per occurrence, not a copy of its description.

Symbol-keyed properties of plain objects are encoded too, which Object.keys does not see. Only enumerable own symbol keys are included, matching what Object.keys does for strings. Set includeSymbolKeys: false to skip them.

Well-known symbols written by a newer runtime that this one does not have are decoded as a descriptive unique symbol and reported through errorMessage, rather than failing the whole decode.

Note that stripKeys does not strip symbol keys — their descriptions still land in the string section. If concealing them matters, use includeSymbolKeys: false alongside stripKeys.

JamPak Mapping Table

The following table shows how JavaScript values are mapped to JamPak formats.

| Source Value | Head Byte | Desc | | --------------------- | ------------------------ | --------------------- | | number | 0x00 - 0x7F, 0xE0 - 0xFF | Small values saved directly, same as MessagePack | | Object | 0x80 - 0x8F, 0xC7 - 0xC9 | Always as Record<string, unknown> | | Array | 0x90 - 0x9F, 0xDA - 0xDC | Array | | string* | 0xB0 - 0xBF, 0xD7 - 0xD9 | Strings are saved in their own unique way in the seporate string section of the data. The only data saved in the value section is the index to the string in the string section. | | keys* | 0xA0 - 0xAF, 0xD4 - 0xD6 | Just like strings above but the data here is just an index to an array that is NOT saved with the file. The object keysArray of the JPEncode class must be saved and passed back to the JPDeocde class or the file won't be readable. | | null | 0xC0 | null | | undefined* | 0xC1 | undefined | | boolean (true, false) | 0xC2 or 0xC3 | True or False | | number (float) | 0xCA or 0xCB | Checks if value needs to be saved as 32 or 64 bit | | number (8-64-bit int) | 0xCC - 0xD3 | numbers between 8 - 64 bit | | bigint* | 0xCF or 0xD3 | Will always be written as 64 bit but will only return as bigint type if outside of safe number range or enforceBigInt is true in options | | Kill byte* | 0xC4 | Triggers the end of the data | | List end* | 0xC5 | Can be useful in extension for splitting data without end decoding process like the kill byte | | Extensions | 0xDD - 0xDF | Uses a secondary index for all built in and user added types |

*Different to MessagePack

JamPak Extension Table

The following are built in types that JamPak works with. Users can add their own Extension Types with numbers between 0x00 - 0xCF. Note: these types are outside of the basic types JSON data deals with so their storage is specific to JamPak.

| Extension Type | Extension Number | Desc | | --------------------------------- | -------------------- | --------------------- | | Map | 0xEE | Just like Object but the keys here are expanded. The size value here are the length of the map, not the buffer. | | Set | 0xEF | Like an Array but with a unique list. The size value here are the size of the set, not the buffer. | | Symbol | 0xF0 | A kind byte followed by an id and/or string reference — see Symbol support. Any generated symbol can also be found in the symbolList array on the JPDecode class. | | RegEx | 0xF1 | Has two strings. Creates new RegExp() | | TypedArray | 0xF2 - 0xFD | BigUint64Array to Uint8ClampedArray | | Buffer | 0xFE | Node default Buffer | | Date | 0xFF | Same function from the example. |

Performance

npm run bench runs JamPak, MessagePack and JSON over the same documents. Every row is verified to round-trip before it is timed. Indicative numbers from Node 24 on x64:

Table-like data, 5,000 rows with repeated keys — what the split string section is built for:

| codec | encode | decode | size | vs JSON | | ----- | ------ | ------ | ---- | ------- | | JSON | 1.32 ms | 1.38 ms | 535 KB | 100% | | JSON + zlib | 3.34 ms | 1.69 ms | 46.4 KB | 8.7% | | JamPak | 1.86 ms | 1.06 ms | 154 KB | 28.9% | | JamPak + zlib | 6.04 ms | 1.30 ms | 47.4 KB | 8.9% | | JamPak + AES (key) | 1.97 ms | 1.13 ms | 155 KB | 28.9% | | MessagePack | 2.68 ms | 4.01 ms | 401 KB | 75.0% |

JamPak decodes this ~4× faster than MessagePack because repeated keys are interned once and resolved by index, rather than re-parsed per row.

Where JamPak does not help: a document of 20,000 unique strings encodes to 103.5% of JSON — the string table has nothing to collapse and costs an index per reference. Reach for compress there (8.6% of JSON), or plain MessagePack.

Numeric data: 50,000 numbers encode in 1.01 ms vs JSON's 2.07 ms, and decode in 0.51 ms vs 1.15 ms, at 83.6% of JSON's size. Compression helps far less on numbers (41.8%) than on text.

Rich types cost nothing extra: a document of Map, Set, Float64Array, Date and bigint encodes in 0.21 ms. MessagePack has no set or typed-array concept, so those come back as Array and Buffer; the JamPak layout preserves them.

Migrating from 1.x

Version 2 is a breaking release. The changes you are most likely to hit:

Files. The header now carries the AES-GCM salt, nonce and tag, and the symbol payload is kind-tagged. 1.x files cannot be read by 2.x and vice versa. Re-encode any archives you need to keep.

Because the magic bytes did not change, 2.x checks the format version explicitly and throws JPVersionError on a 1.x file rather than misreading it. Use inspectHeader to sort a directory of mixed-version files without decoding them:

import { inspectHeader } from "jampak";

const info = inspectHeader(fs.readFileSync(path).subarray(0, 84));

if (!info.supported) console.warn(`${path}: ${info.reason}`);

Dependencies. bireader and cbor-x are gone. The package has no runtime dependencies. bireader 5.x had changed in ways that no longer compiled against this codebase, which is what prompted the rewrite.

Extension API. Extension functions receive JPReader / JPWriter instead of BiReader / BiWriter:

// 1.x
const bw = new BiWriter(Buffer.alloc(12));
bw.endian = encoder.endian;
bw.uint32 = value;
return bw.return() as Buffer;

// 2.x
const bw = new JPWriter({ initialSize: 12, little: encoder.endian !== "big" });
bw.u32(value);
return bw.toBuffer();

Accessors are methods rather than getter/setter properties: r.u32() / w.u32(v) in place of br.uint32 / bw.uint32 = v. Reading is bounds checked and throws JPRangeError past the end.

Errors are thrown. encode used to catch everything, log it and return Buffer.alloc(0); decode returned undefined. Both now throw, because an empty Buffer is indistinguishable from a successfully encoded empty document. Wrap calls in try/catch if you were relying on the old behaviour. Non-fatal problems (CRC mismatch, an unresolvable string index) still go to errored / errorMessage, and are no longer printed to the console.

Encryption is a different model. Files written by 1.x used a 32-bit-seeded key with a key-derived IV and no authentication. 2.x uses AES-256-GCM with a per-file random salt and nonce.

The encryptionKey option is gone — the 32-bit seed does not exist any more. In its place:

| 1.x | 2.x | | --- | --- | | encryptionKey: number (option) | key: Buffer (32 bytes) or password: string | | encryptionKey: number (property) | encryptionKey: Buffer — the actual AES key used | | stripEncryptKey hides a 32-bit seed | stripEncryptKey withholds the 32-byte key; it is now the only switch between self-opening and secret |

Supplying a password no longer implies stripping. Set stripEncryptKey: true explicitly if only you should be able to open the file — the encoder warns on errorMessage if you supply a secret without it.

msgpack: true now produces MessagePack. In 1.x this option delegated to cbor-x, which emits CBOR — the payload was not readable by any MessagePack implementation despite the option name. It is now a real MessagePack encoder.

Symbol keys are encoded by default. Objects with symbol-keyed enumerable properties produce slightly larger files and now round-trip those properties. Set includeSymbolKeys: false for the old behaviour.

BiReader-era growthIncrement was the initial allocation and the growth step, defaulting to 16 MB — so encoding {foo:"bar"} allocated 32 MB across the two writers. It is now the growth cap only; use initialSize (default 4096) for the up-front allocation.

Prerequisites

This is a universal JavaScript library that supports only NodeJS. NodeJS v18 is required.

Development

npm run typecheck   # tsc --noEmit over src
npm test            # compile to build/ and run node:test
npm run bench       # micro-benchmarks
npm run build       # rollup bundles into dist/

Binary template

For a full understanding of the file structure, the most up-to-date JAMPAK.bt binary template can be found here.

License

This software uses the ISC license:

https://opensource.org/licenses/ISC

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

JamPak for Node/JavaScript/TypeScript

How it works

Synopsis

Table of Contents

Install

API

new JPEncode(EncoderOptions?)

new JPEncodeAsync(EncoderOptions?)

EncoderOptions

Class JPEncode functions

Class JPEncode objects

new JPDecode(DecoderOptions?)

new JPDecodeAsync(DecoderOptions?)

DecoderOptions

Types to JSON table

Class JPDecode functions

Class JPDecode objects

inspectHeader(buffer)

Format versions

Extension Types

ExtensionCodec context

Advanced Features

How the AES key works

The two modes

What is on disk

Limits and hardening

JamPak Specification

Symbol support

JamPak Mapping Table

JamPak Extension Table

Performance

Migrating from 1.x

Prerequisites

Development

Binary template

License

`new JPEncode(EncoderOptions?)`

`new JPEncodeAsync(EncoderOptions?)`

`EncoderOptions`

Class `JPEncode` functions

Class `JPEncode` objects

`new JPDecode(DecoderOptions?)`

`new JPDecodeAsync(DecoderOptions?)`

`DecoderOptions`

Class `JPDecode` functions

Class `JPDecode` objects

`inspectHeader(buffer)`