datapack

Datapack is a JS library that provide high performance methods pack and unpack binary data using schema of data model. This library can be used in both NodeJS and Browser environtment.

Installation

npm install datapack

Usage

Simple value

import { pack, unpack, UINT8 } from "datapack";

const packed = pack(100, UINT8);
const unpacked = unpack(packed, UINT8);

console.log(unpacked); // 100

Array

import { pack, unpack, UINT8, INT16 } from "datapack";

const schema = [UINT8, INT16];
const value = [100, 200, 50];
const packed = pack(value, schema);
const unpacked = unpack(packed, schema);

console.log(unpacked); // [100, 200, 50]

Object

import { pack, unpack, UINT16, INT8 } from "datapack";

const schema = {
  aaa: [UINT16],
  obj1: {
    obj11: UINT16,
    obj4: [INT8],
  },
};
const value = {
  aaa: [1],
  obj1: {
    obj11: 2,
    obj4: [2],
  },
};
const packed = pack(value, schema);
const unpacked = unpack(packed, schema);

console.log(unpacked);

TypeScript Support

datapack is written in TypeScript and comes with first-class type support.

Dynamic Type Generation

One of the most powerful features of datapack is its ability to generate TypeScript types dynamically from your schema. When you use the unpack function, the return type is automatically inferred from the schema you provide. This gives you full type safety and autocompletion for your unpacked data, eliminating guesswork and reducing runtime errors.

Here's an example of how it works:

import { pack, unpack, UINT32, STRING, BOOL, UINT8 } from "datapack";

const profileSchema = {
  userId: UINT32,
  nickName: STRING,
  isVip: BOOL,
  age: UINT8,
};

const profileData = {
  userId: 101,
  nickName: "Alice",
  isVip: true,
  age: 34,
};

const packedProfile = pack(profileData, profileSchema);

// The 'unpackedProfile' variable will have a fully typed structure:
// {
//   userId: number;
//   nickName: string;
//   isVip: boolean;
//   age: number;
// }
const unpackedProfile = unpack(packedProfile, profileSchema);

// You get autocompletion and type checking!
console.log(unpackedProfile.nickName.toUpperCase()); // Works!
// console.log(unpackedProfile.invalidProperty); // TypeScript error!

Complex data structures

import { pack, unpack, UINT32, STRING, BOOL, UINT8, UINT16 } from "datapack";

const profileSchema = {
  userId: UINT32,
  nickName: STRING,
  isVip: BOOL,
  age: UINT8,
};

const stateDataSchema = {
  users: [profileSchema],
  posts: [
    {
      postId: UINT32,
      title: STRING,
      score: UINT16,
      authors: [profileSchema],
    },
  ],
};

const stateData = {
  users: [
    {
      userId: 101,
      nickName: "ABC",
      isVip: true,
      age: 34,
    },
  ],
  posts: [
    {
      postId: 100,
      title: "Hello World!",
      score: 999,
      authors: [
        {
          userId: 102,
          nickName: "DEF",
          isVip: false,
          age: 28,
        },
      ],
    },
  ],
};

const packedState = pack(stateData, stateDataSchema);
const unpackedState = unpack(packedState, stateDataSchema);

console.log(unpackedState);

Options

The pack and unpack functions accept an optional options object that allows you to enable checksum validation and encryption.

import { pack, unpack, UINT8, STRING } from "datapack";

const schema = {
  a: UINT8,
  b: STRING,
};
const data = {
  a: 255,
  b: "test string",
};
const options = {
  useCheckSum: true,
  useEncrypt: true,
  secret: 123,
};

const packed = pack(data, schema, options);
const unpacked = unpack(packed, schema, options);

console.log(unpacked);

All Data Types

import {
  pack,
  unpack,
  UINT8,
  INT8,
  UINT16,
  INT16,
  INT32,
  BOOL,
  STRING,
  OBJECT,
  UINT64,
  INT64,
  BINARY,
  UINT32,
  FLOAT,
} from "datapack";
import { Buffer } from "buffer";

// UINT8
const packed_UINT8 = pack(100, UINT8);
const unpacked_UINT8 = unpack(packed_UINT8, UINT8);
console.log(unpacked_UINT8); // 100

// INT8
const packed_INT8 = pack(100, INT8);
const unpacked_INT8 = unpack(packed_INT8, INT8);
console.log(unpacked_INT8); // 100

// UINT16
const packed_UINT16 = pack(10000, UINT16);
const unpacked_UINT16 = unpack(packed_UINT16, UINT16);
console.log(unpacked_UINT16); // 10000

// INT16
const packed_INT16 = pack(4000, INT16);
const unpacked_INT16 = unpack(packed_INT16, INT16);
console.log(unpacked_INT16); // 4000

// INT32
const packed_INT32 = pack(2000000000, INT32);
const unpacked_INT32 = unpack(packed_INT32, INT32);
console.log(unpacked_INT32); // 2000000000

// FLOAT
const packed_FLOAT = pack(123.456, FLOAT);
const unpacked_FLOAT = unpack(packed_FLOAT, FLOAT);
console.log(unpacked_FLOAT); // 123.456

// BOOL
const packed_BOOL_true = pack(true, BOOL);
const unpacked_BOOL_true = unpack(packed_BOOL_true, BOOL);
console.log(unpacked_BOOL_true); // true

// STRING
const packed_STRING = pack("abc", STRING);
const unpacked_STRING = unpack(packed_STRING, STRING);
console.log(unpacked_STRING); // "abc"

// OBJECT
const packed_OBJECT = pack({ abc: 100 }, OBJECT);
const unpacked_OBJECT = unpack(packed_OBJECT, OBJECT);
console.log(unpacked_OBJECT); // { abc: 100 }

// UINT64
const packed_UINT64 = pack(BigInt("10000000000"), UINT64);
const unpacked_UINT64 = unpack(packed_UINT64, UINT64);
console.log(unpacked_UINT64); // 10000000000n

// INT64
const packed_INT64 = pack(BigInt("-10000000000"), INT64);
const unpacked_INT64 = unpack(packed_INT64, INT64);
console.log(unpacked_INT64); // -10000000000n

// BINARY
const packed_BINARY = pack(Buffer.from("abc"), BINARY);
const unpacked_BINARY = unpack(packed_BINARY, BINARY);
console.log(unpacked_BINARY); // <Buffer 61 62 63>

Benchmark

Host Specs:

• CPU: AMD EPYC 7B12 @ 2.25GHz (2 cores)
• RAM: 8GB
• OS: Linux

Results:

| Scenario | Datapack (pack) | Datapack (unpack) | JSON (stringify) | JSON (parse) | |---|---|---|---|---| | Simple object (number fields) | ~626,680 ops/sec | ~666,401 ops/sec | ~653,272 ops/sec | ~819,183 ops/sec | | Complex object | ~39,577 ops/sec | ~44,942 ops/sec | ~57,491 ops/sec | ~55,389 ops/sec | | Big object (~1MB) | ~18 ops/sec | ~16 ops/sec | ~25 ops/sec | ~22 ops/sec |

Packed Size Comparison

| Scenario | Datapack Size | JSON Size | |---|---|---| | Simple object (number fields) | 20 bytes | 82 bytes | | Complex object | 530 bytes | 1731 bytes | | Big object (~1MB) | 1,300,010 bytes | 4,275,021 bytes |

Test coverage

-------------|---------|----------|---------|---------|------------------- File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s -------------|---------|----------|---------|---------|------------------- All files | 100 | 100 | 100 | 100 |
index.ts | 100 | 100 | 100 | 100 |
packer.ts | 100 | 100 | 100 | 100 |
unpacker.ts | 100 | 100 | 100 | 100 |
utils.ts | 100 | 100 | 100 | 100 |
-------------|---------|----------|---------|---------|-------------------

Compatibility

This library can be used in both NodeJS and Browser environtment.

License

MIT