JSON Structured Transfer Protocol

The JSON Structured Transfer Protocol efficiently transfers JSON data by separating schema (field structure) from values, using MessagePack encoding to reduce bandwidth and improve reliability on unstable networks.

Features

• Schema-based encoding: Separates field names from values for efficient transmission
• MessagePack encoding: Binary format for compact data transfer
• State machine: Strict protocol state management following SPECS 11.8
• Deterministic Schema IDs: FNV-1a hash for consistent schema identification
• Error recovery: Handles E100 (unknown schema), E200 (decode error), E300 (invalid payload)
• Type-safe: Full TypeScript support with strict type checking

Performance

Comparison of serialization methods across various data structures:

| Method | Avg Size | vs JSON | Avg Time | vs JSON | | ------------- | -------- | ------- | -------- | ------- | | JSON | 177B | - | 0.95ms | - | | MessagePack | 134B | 24.3% | 1.21ms | +28.4% | | This Protocol | 102B | 42.4% | 0.95ms | +0.8% |

Key Insights:

• Best size reduction: This protocol achieves 42.4% smaller payload than JSON
• Near-zero overhead: Only 0.8% slower than native JSON (essentially the same speed)
• Faster than MessagePack: 21% faster than MessagePack while achieving better compression
• Highly optimized: Flat array wire format, JIT-compiled codecs, reusable buffers, and shared instances
• Use case: Ideal for bandwidth-constrained environments where both size and speed matter

Performance by Data Type:

• Simple objects: 25% overhead (67% smaller payloads)
• User profiles: 52% overhead (55% smaller payloads)
• Nested objects: 7% overhead (30% smaller payloads)
• Array of objects: -0.3% overhead (faster than JSON, 38% smaller payloads)

Test Conditions:

• 1,000 iterations per method (with V8 JIT warmup)
• 4 different data structures tested
• Tests include simple objects, nested structures, and arrays
• Schema registration overhead included for this protocol

Run pnpm benchmark to reproduce results on your machine.

Installation

npm install json-structured-transfer-protocol
# or
pnpm install json-structured-transfer-protocol
# or
yarn add json-structured-transfer-protocol

Quick Start

Basic Usage

import { ProtocolSession, SessionState, MessageType } from 'json-structured-transfer-protocol';

// Create a session
const session = new ProtocolSession({
  onMessage: (message) => {
    console.log('Outgoing message:', message);
    // Send this message to the remote peer
  },
  onData: (data) => {
    console.log('Received data:', data);
  },
  onError: (error) => {
    console.error('Protocol error:', error);
  },
});

// Complete handshake
await session.onMessage({ type: MessageType.HELLO, version: '0.1' });
await session.onMessage({ type: MessageType.HELLO_ACK, version: '0.1' });
await session.onMessage({ type: MessageType.SCHEMA_ACK, schema_id: 123 });

// Send data
const data = { name: 'Alice', age: 30 };
await session.send(data);

Two-Peer Communication

import { ProtocolSession, MessageType } from 'json-structured-transfer-protocol';

// Peer 1
const peer1 = new ProtocolSession({
  onMessage: async (msg) => {
    // Send to peer2
    await peer2.onMessage(msg);
  },
  onData: (data) => console.log('Peer1 received:', data),
});

// Peer 2
const peer2 = new ProtocolSession({
  onMessage: async (msg) => {
    // Send to peer1
    await peer1.onMessage(msg);
  },
  onData: (data) => console.log('Peer2 received:', data),
});

// Initiate handshake from peer2
await peer2.onMessage({ type: MessageType.HELLO, version: '0.1' });
await peer1.onMessage({ type: MessageType.HELLO, version: '0.1' });

// Send data from peer1 to peer2
await peer1.send({ message: 'Hello from peer1!' });

API Reference

ProtocolSession

The main class for managing protocol sessions.

Constructor

new ProtocolSession(config?: SessionConfig)

SessionConfig:

• onMessage?: (message: ProtocolMessage) => void | Promise<void> - Called when a message should be sent
• onData?: (data: Record<string, unknown>) => void | Promise<void> - Called when data is received
• onError?: (error: Error) => void | Promise<void> - Called when an error occurs

Methods

send(json: Record<string, unknown>): Promise<void>

Send JSON data as a PAYLOAD message. Automatically sends SCHEMA_DEF if the schema is not registered.

await session.send({ name: 'Alice', age: 30 });

onMessage(message: ProtocolMessage): Promise<void>

Process an incoming protocol message.

await session.onMessage({
  type: MessageType.PAYLOAD,
  payload: { schema_id: 12345, values: ['Alice', 30] },
});

getState(): SessionState

Get the current session state.

const state = session.getState();
console.log(state); // 'S3_STEADY'

close(): Promise<void>

Close the session by sending a GOODBYE message.

await session.close();

reset(): void

Reset the session to initial state.

session.reset();

Session States

The protocol follows these states (SPECS 11.8):

• S0_INIT - Initial state
• S1_HANDSHAKE - Handshake in progress
• S2_SCHEMA_SYNC - Schema synchronization
• S3_STEADY - Normal operation (data exchange)
• S4_RESYNC - Resynchronization after error
• S6_CLOSED - Session closed

Message Types

• HELLO - Initiate handshake
• HELLO_ACK - Acknowledge handshake
• SCHEMA_DEF - Define a schema
• SCHEMA_ACK - Acknowledge schema
• SCHEMA_REQ - Request missing schema
• PAYLOAD - Transfer data
• ERROR - Report error
• RESYNC_START - Start resynchronization
• RESYNC_END - End resynchronization
• GOODBYE - Close session

Error Codes

• E100_UNKNOWN_SCHEMA - Schema not found in registry
• E200_DECODE_ERROR - Failed to decode MessagePack data
• E300_INVALID_PAYLOAD - Invalid payload structure

Advanced Usage

Manual Schema Management

import { SchemaRegistry, Encoder, Decoder } from 'json-structured-transfer-protocol';

const registry = new SchemaRegistry();
const encoder = new Encoder();
const decoder = new Decoder(registry);

// Generate and register schema
const json = { name: 'Alice', age: 30 };
const schema = encoder.generateSchemaDefinition(json);
registry.register(schema);

// Encode to payload
const payload = encoder.encode(json);
console.log(payload); // { schema_id: 123456, values: ['Alice', 30] }

// Decode back to JSON
const decoded = decoder.decode(payload);
console.log(decoded); // { name: 'Alice', age: 30 }

MessagePack Encoding

import { Encoder, Decoder, SchemaRegistry } from 'json-structured-transfer-protocol';

const encoder = new Encoder();
const registry = new SchemaRegistry();
const decoder = new Decoder(registry);

// Encode to MessagePack binary
const json = { name: 'Alice', age: 30 };
const schema = encoder.generateSchemaDefinition(json);
registry.register(schema);

const buffer = encoder.encodeToMessagePack(json);
console.log(buffer); // Buffer <...>

// Decode from MessagePack binary
const decoded = decoder.decodeFromMessagePack(buffer);
console.log(decoded); // { name: 'Alice', age: 30 }

Error Handling

import { ProtocolSession, ProtocolError, ErrorCode } from 'json-structured-transfer-protocol';

const session = new ProtocolSession({
  onError: (error) => {
    if (error instanceof ProtocolError) {
      switch (error.code) {
        case ErrorCode.E100_UNKNOWN_SCHEMA:
          console.log('Missing schema:', error.schemaId);
          // Schema will be requested automatically
          break;
        case ErrorCode.E200_DECODE_ERROR:
          console.error('Decode failed:', error.message);
          break;
        case ErrorCode.E300_INVALID_PAYLOAD:
          console.error('Invalid payload:', error.message);
          break;
      }
    }
  },
});

Protocol Flow

Handshake

Peer A (S0_INIT)                    Peer B (S0_INIT)
     |                                      |
     |            HELLO                     |
     |------------------------------------->|
     |                               (S1_HANDSHAKE)
     |          HELLO_ACK                   |
     |<-------------------------------------|
(S2_SCHEMA_SYNC)                            |
     |         SCHEMA_ACK                   |
     |------------------------------------->|
(S3_STEADY)                           (S3_STEADY)

Data Exchange

Peer A (S3_STEADY)                  Peer B (S3_STEADY)
     |                                      |
     |         SCHEMA_DEF                   |
     |------------------------------------->|
     |          SCHEMA_ACK                  |
     |<-------------------------------------|
     |           PAYLOAD                    |
     |------------------------------------->|
     |                                 (receives data)

Error Recovery

Peer A (S3_STEADY)                  Peer B (S3_STEADY)
     |                                      |
     |    PAYLOAD (unknown schema)          |
     |------------------------------------->|
     |          SCHEMA_REQ                  |
     |<-------------------------------------|
(S2_SCHEMA_SYNC)                            |
     |         SCHEMA_DEF                   |
     |------------------------------------->|
(S3_STEADY)                                 |

Testing

# Run tests
pnpm test

# Run tests in watch mode
pnpm test:watch

# Build project
pnpm build

# Type check
pnpm typecheck

Development

# Install dependencies
pnpm install

# Run tests
pnpm test

# Build
pnpm build

Documentation

• Publishing Guide - How to publish to npm
• Workflow Documentation - GitHub Actions workflows
• Setup Summary - Quick setup reference

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.