BinSchema

Binary protocol schema definition and code generation tool. Define binary formats in JSON5, generate type-safe parsers and serializers for TypeScript, Python, Go, Rust, and Zig.

Features

• Bit-level precision - 1-64 bit fields with configurable bit ordering (MSB/LSB)
• Rich type system - Primitives, booleans, strings (UTF-8/ASCII/Latin-1/UTF-16), byte arrays, discriminated unions, back-references
• Computed fields - Auto-calculate lengths, positions, checksums, and counts
• Multi-language - Generate TypeScript, Python, Go, Rust, and Zig from a single schema
• Cross-language testing - JSON test format validates all implementations produce identical output

Installation

# Install dependencies
npm install

# Build
npm run build

# Link globally (optional)
npm link

CLI Usage

# Show help
binschema help
binschema help generate
binschema help docs

# Generate TypeScript code
binschema generate --language ts --schema examples/dns.schema.json --out ./generated

# Generate Go code
binschema generate --language go --schema examples/dns.schema.json --out ./generated

# Generate Rust code
binschema generate --language rust --schema examples/dns.schema.json --out ./generated

# Generate Python code
binschema generate --language python --schema examples/dns.schema.json --out ./generated

# Generate Zig code
binschema generate --language zig --schema examples/dns.schema.json --out ./generated

# Generate HTML documentation
binschema docs build --schema examples/zip.schema.json --out docs.html

Schema Format

Schemas are defined in JSON5 format:

{
  "config": {
    "endianness": "big_endian",  // or "little_endian"
    "bit_order": "msb_first"     // or "lsb_first"
  },
  "types": {
    "MessageHeader": {
      "sequence": [
        { "name": "version", "type": "uint8" },
        { "name": "flags", "type": "uint8" },
        { "name": "length", "type": "uint16", "computed": { "type": "length_of", "target": "payload" } },
        { "name": "payload", "type": "array", "kind": "field_referenced", "length_field": "length", "items": { "type": "uint8" } }
      ]
    }
  }
}

Primitive Types

| Type | Size | Description | |------|------|-------------| | bool | 1 byte | Boolean (0x00 = false, 0x01 = true) | | uint8, int8 | 1 byte | 8-bit integers | | uint16, int16 | 2 bytes | 16-bit integers | | uint32, int32 | 4 bytes | 32-bit integers | | uint64, int64 | 8 bytes | 64-bit integers | | float32 | 4 bytes | IEEE 754 single precision | | float64 | 8 bytes | IEEE 754 double precision | | bitfield | variable | Bit-level fields (1-64 bits) | | bytes | variable | Byte array (sugar for array<uint8>) |

Strings

Strings support four encodings: utf8 (default), ascii, latin1, and utf16. UTF-16 strings use 2-byte code units with endianness from the field or global config.

Bytes

The bytes type is shorthand for array<uint8> — no items field needed:

{ "name": "data", "type": "bytes", "kind": "length_prefixed", "length_type": "uint16" }

Supports all array kinds: fixed, length_prefixed, field_referenced, null_terminated, eof_terminated, etc.

Array Kinds

| Kind | Description | |------|-------------| | fixed | Fixed-length array (length property) | | length_prefixed | Count prefix before elements (length_type property) | | byte_length_prefixed | Byte-length prefix (ASN.1/DER style) | | field_referenced | Length from earlier field (length_field property) | | null_terminated | Read until 0x00 byte | | variant_terminated | Read until terminal variant type (terminal_variants property) | | eof_terminated | Read until end of stream |

Array Transforms

The optional transform slot rewrites how array elements are laid out on the wire while leaving the logical values untouched. It is orthogonal to kind (which only controls length framing) and to the item's own wire type.

| Transform | Description | |-----------|-------------| | delta | Store each element as the difference from the previous one (value[i] - value[i-1], first relative to 0). Shrinks sorted/correlated integer columns — timestamps, IDs, sensor data. |

delta is a pure wire transform: the decoded array is identical to the input (the deltas exist only on the wire). The delta is written using the item's own encoding, so a zigzag varlength item yields zigzag-varint deltas — the right default, since deltas go negative when the data dips. On a plain unsigned fixed-width item it only works for monotonic data (author's responsibility, like Thrift). Items must be numeric (uint*/int*) or varlength. v1 is whole-array with a single 64-bit accumulator.

// Timestamps [1700000000, 1700000001, 1700000003] are stored as deltas
// [1700000000, 1, 2] — only the first element pays the full width.
{
  "name": "timestamps",
  "type": "array",
  "kind": "length_prefixed",
  "length_type": "uint16",
  "transform": "delta",
  "items": { "type": "varlength", "encoding": "leb128" }
}

// Signed deltas: zigzag handles values that dip below the previous one.
{
  "name": "samples",
  "type": "array",
  "kind": "length_prefixed",
  "length_type": "uint16",
  "transform": "delta",
  "items": { "type": "varlength", "encoding": "zigzag" }
}

Compressed Regions

The compressed wrapper type marks a byte region whose wire bytes are not the logical value — they are the inner type run through a codec. It is the region-level analogue of the array delta transform: where delta rewrites array elements, compressed transforms a whole byte region. On encode the inner value_type is serialized to a buffer, compressed, and framed; on decode the bytes are decompressed and the inner type is parsed back out.

Wire layout:

[uncompressed_size: size_type][compressed_length: length_type][compressed_bytes…]

Both size fields default to uint32 and are configurable via size_type / length_type. They are consumed framing (like an array length prefix) — they do not appear in the decoded value, so the logical value is just the inner type on both sides (value === decoded_value).

store, deflate, and gzip are built-in codecs that work after generate with no extra wiring — each runtime uses its language's standard library (Go compress/flate+compress/gzip, Python zlib+gzip, Rust flate2, TypeScript vendors fflate). store is the identity codec (compressed bytes == inner bytes), useful when you want the framing without paying for compression.

// A length-prefixed compressed blob that decodes as a SignalBatch.
{
  "name": "payload",
  "type": "compressed",
  "codec": "deflate",
  "value_type": "SignalBatch"
}

// Narrow the framing to uint16 size + uint16 length prefixes.
{
  "name": "payload",
  "type": "compressed",
  "codec": "store",
  "value_type": "Inner",
  "size_type": "uint16",
  "length_type": "uint16"
}

The codec registry is pluggable: unknown codec names (e.g. zstd, lz4, snappy) resolve to a codec you register before encoding/decoding, so you can plug in any compression library without forcing a heavy dependency on everyone:

// TypeScript
import { registerCodec } from "./binschema_runtime/codecs.js";
registerCodec("zstd", { compress: zstdCompress, decompress: zstdDecompress });

# Python
from binschema_runtime import register_codec, Codec
register_codec("zstd", MyZstdCodec())

// Go
runtime.RegisterCodec("zstd", myZstdCodec{})

// Rust
binschema_runtime::register_codec("zstd", std::sync::Arc::new(MyZstdCodec));

Computed Fields

// Auto-compute length of another field
{ "name": "length", "type": "uint16", "computed": { "type": "length_of", "target": "data" } }

// Auto-compute position/offset to another field
{ "name": "offset", "type": "uint32", "computed": { "type": "position_of", "target": "data" } }

// Auto-compute element count
{ "name": "count", "type": "uint8", "computed": { "type": "count_of", "target": "items" } }

// Auto-compute CRC32 checksum
{ "name": "checksum", "type": "uint32", "computed": { "type": "crc32_of", "target": "data" } }

Discriminated Unions

{
  "type": "discriminated_union",
  "discriminator": { "peek": "uint8" },  // Peek at discriminator without consuming
  "variants": [
    { "type": "TextMessage", "when": "value === 0x01" },
    { "type": "BinaryMessage", "when": "value === 0x02" }
  ]
}

Streaming

For protocols that frame their data as a length-prefixed array of records — chat messages, telemetry samples, log lines arriving over a WebSocket or fetch ReadableStream — BinSchema can generate an async-generator decoder that yields one item at a time as bytes arrive, without buffering the whole array in memory.

Enable streaming codegen with the generate_streaming option:

import { generateTypeScript } from "binschema";

const code = generateTypeScript(schema, { generate_streaming: true });

For every top-level type that consists of exactly one length-prefixed array (length_prefixed or length_prefixed_items kind), the generator emits an extra function alongside the synchronous decoder class:

// Synchronous (existing):
const decoder = new MessageArrayDecoder(allBytes);
const result = decoder.decode();  // requires the entire array up front

// Streaming (new):
for await (const message of decodeMessageArrayStream(reader)) {
  handle(message);  // first item arrives as soon as enough bytes have buffered
}

reader is a ReadableStreamDefaultReader<Uint8Array> — obtain one from fetch().body.getReader(), a WebSocket adapter, or any source that produces Uint8Array chunks.

When to use streaming vs sync

• Sync (new XDecoder(bytes).decode()) — the data is already in memory or small enough that latency to first byte doesn't matter. Fastest per-item.
• Streaming (decodeXStream(reader)) — the array is large, the source is chunked (network, file pipe), or you want to start processing items before the tail arrives. Yields with for await so a slow consumer naturally applies backpressure to the underlying reader.

Mechanism

• length_prefixed_items arrays (per-item byte-length prefix on the wire): the streaming layer reads the prefix, accumulates exactly that many bytes from the reader, then decodes the item over the slice. No speculation.
• length_prefixed arrays (no per-item prefix): the streaming layer speculatively decodes each item against the current buffer. On BinSchemaError(INCOMPLETE_DATA) it pulls another chunk and retries from the same buffer position. Any other error code is fatal and propagates with per-item context.

Error codes

The streaming layer throws BinSchemaError instances with a .code field drawn from ErrorCode:

| Code | Meaning | |-----------------------|----------------------------------------------------------| | INCOMPLETE_DATA | Stream ended before all items / item bytes were read | | INVALID_VALUE | A read returned an out-of-range value | | INVALID_ENCODING | Wire-format invariant violated (e.g. bad bitfield width) | | INVALID_UTF8 | A length-prefixed string was not valid UTF-8 | | INVALID_VARIANT | Discriminated-union discriminator didn't match any arm | | ALIGNMENT_REQUIRED | Byte-aligned read attempted at a non-zero bit offset | | OUT_OF_BOUNDS | Seek/peek past buffer end |

Network errors from the underlying reader.read() propagate unchanged so callers can distinguish "decoder gave up" from "socket died". Errors carry an .context string with the item index ("item 3/100") and a .position byte offset where applicable.

Examples

See the examples/ directory for complete schemas:

• DNS Protocol (dns.schema.json) - Domain name compression, bitfields, variable-length labels
• ZIP Format (zip.schema.json) - File headers, computed CRCs, random-access offsets
• PNG Format (png.schema.json) - Chunk-based format with variant-terminated arrays
• MIDI Format (midi.schema.json) - Variable-length quantities (VLQ), track chunks

Development

# Run tests
npm test

# Run specific tests
npm test -- --filter=dns

# Debug tests
DEBUG_TEST=1 npm test -- --filter=dns

# Watch mode
npm run watch

Project Structure

binschema/
  src/
    schema/        # Zod schemas and validation
    generators/    # Code generators (TypeScript, Python, Go, Rust, Zig, HTML)
    runtime/       # Reference encoder/decoder implementation
    tests/         # Test suites
    cli/           # Command-line interface
  examples/        # Example schemas
  go/              # Go implementation
  rust/            # Rust implementation (experimental)

License

MIT